pre-3.16 regression on open() scalability
Hi Paul,
I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop:
At about 50 cores worth of processes, 3.15 and the pre-3.16 code start to diverge, with 3.15 scaling better:
http://sr71.net/~dave/intel/3.16-open1regression-0.png
Some profiles point to a big increase in contention inside slub.c's get_partial_node() (the allocation side of the slub code) causing the regression. That particular open() test is known to do a lot of slab operations. But, the odd part is that the slub code hasn't been touched much.
So, I bisected it down to this:
Quoted 5 lines
commit
ac1bea85781e9004da9b3e8a4b097c18492d857cAuthor: Paul E. McKenney <[email protected]> Date: Sun Mar 16 21:36:25 2014 -0700sched,rcu: Make
cond_resched()report RCU quiescent states
Specifically, if I raise RCU_COND_RESCHED_LIM, things get back to their 3.15 levels.
Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
The top RCU functions in the profiles are as follows:
Quoted 9 lines
3.15.0-xxx: 2.58% open1_processes [kernel.kallsyms] [k] file_free_rcu 3.15.0-xxx: 2.45% open1_processes [kernel.kallsyms] [k] __d_lookup_rcu 3.15.0-xxx: 2.41% open1_processes [kernel.kallsyms] [k] rcu_process_callbacks 3.15.0-xxx: 1.87% open1_processes [kernel.kallsyms] [k] __call_rcu.constprop.103.16.0-rc0: 2.68% open1_processes [kernel.kallsyms] [k] rcu_process_callbacks 3.16.0-rc0: 2.68% open1_processes [kernel.kallsyms] [k] file_free_rcu 3.16.0-rc0: 1.55% open1_processes [kernel.kallsyms] [k] __call_rcu.constprop.10 3.16.0-rc0: 1.28% open1_processes [kernel.kallsyms] [k] __d_lookup_rcu
With everything else equal, we'd expect to see all of these _higher_ in the profiles on a the faster kernel (3.15) since it has more RCU work to do.
But, they're all _roughly_ the same. __d_lookup_rcu went up in the profile on the fast one (3.15) probably because there _were_ more lookups happening there.
rcu_process_callbacks makes me syspicious. It went up slightly (probably in the noise), but it _should_ have dropped due to there being less RCU work to do.
This supports the theory that there are more callbacks happening than before, causing more slab lock contention, which is the actual trigger for the performance drop.
I also hacked in an interface to make RCU_COND_RESCHED_LIM a tunable. Making it huge instantly makes my test go fast, and dropping it to 256 instantly makes it slow. Some brief toying with it shows that RCU_COND_RESCHED_LIM has to be about 100,000 before performance gets back to where it was before.
Quoted 25 lines · Dave Hansen
Hi Paul,
I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop:
At about 50 cores worth of processes, 3.15 and the pre-3.16 code start to diverge, with 3.15 scaling better:
http://sr71.net/~dave/intel/3.16-open1regression-0.pngSome profiles point to a big increase in contention inside slub.c's
get_partial_node()(the allocation side of the slub code) causing the regression. That particularopen()test is known to do a lot of slab operations. But, the odd part is that the slub code hasn't been touched much.So, I bisected it down to this:
Quoted 4 lines
commit
ac1bea85781e9004da9b3e8a4b097c18492d857cAuthor: Paul E. McKenney <[email protected]> Date: Sun Mar 16 21:36:25 2014 -0700
Quoted 8 lines
Quoted 1 line
sched,rcu: Make
cond_resched()report RCU quiescent statesSpecifically, if I raise
RCU_COND_RESCHED_LIM, things get back to their 3.15 levels.Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is rcu_preempt for CONFIG_PREEMPT and rcu_sched for !CONFIG_PREEMPT.
Another possibility is that someone is invoking cond_reched() in an incredibly tight loop.
Quoted 27 lines
The top RCU functions in the profiles are as follows:
Quoted 9 lines
3.15.0-xxx: 2.58% open1_processes [kernel.kallsyms] [k] file_free_rcu 3.15.0-xxx: 2.45% open1_processes [kernel.kallsyms] [k] __d_lookup_rcu 3.15.0-xxx: 2.41% open1_processes [kernel.kallsyms] [k] rcu_process_callbacks 3.15.0-xxx: 1.87% open1_processes [kernel.kallsyms] [k] __call_rcu.constprop.103.16.0-rc0: 2.68% open1_processes [kernel.kallsyms] [k] rcu_process_callbacks 3.16.0-rc0: 2.68% open1_processes [kernel.kallsyms] [k] file_free_rcu 3.16.0-rc0: 1.55% open1_processes [kernel.kallsyms] [k] __call_rcu.constprop.10 3.16.0-rc0: 1.28% open1_processes [kernel.kallsyms] [k] __d_lookup_rcuWith everything else equal, we'd expect to see all of these _higher_ in the profiles on a the faster kernel (3.15) since it has more RCU work to do.
But, they're all _roughly_ the same.
__d_lookup_rcuwent up in the profile on the fast one (3.15) probably because there _were_ more lookups happening there.
rcu_process_callbacksmakes me syspicious. It went up slightly (probably in the noise), but it _should_ have dropped due to there being less RCU work to do.This supports the theory that there are more callbacks happening than before, causing more slab lock contention, which is the actual trigger for the performance drop.
Quoted 5 lines
I also hacked in an interface to make
RCU_COND_RESCHED_LIMa tunable. Making it huge instantly makes my test go fast, and dropping it to 256 instantly makes it slow. Some brief toying with it shows thatRCU_COND_RESCHED_LIMhas to be about 100,000 before performance gets back to where it was before.
That is way bigger than I would expect. My bet is that someone is invoking cond_resched() in a 10s-of-nanoseconds tight loop.
But please feel free to send along your patch, CCing LKML. Longer term, I probably need to take a more algorithmic approach, but what you have will be useful to benchmarkers until then.
Thanx, Paul
Quoted 16 lines · Paul E. McKenney
On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dav
Quoted 6 lines
So, I bisected it down to this:
Quoted 4 lines
commit
ac1bea85781e9004da9b3e8a4b097c18492d857cAuthor: Paul E. McKenney <[email protected]> Date: Sun Mar 16 21:36:25 2014 -0700Quoted 8 lines
Quoted 1 line
sched,rcu: Make
cond_resched()report RCU quiescent statesSpecifically, if I raise
RCU_COND_RESCHED_LIM, things get back to their 3.15 levels.Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quoted 7 lines
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is
rcu_preemptforCONFIG_PREEMPTandrcu_schedfor !CONFIG_PREEMPT.Another possibility is that someone is invoking
cond_reched()in an incredibly tight loop.
open() does at least a couple of allocations in getname(), get_empty_filp() and apparmor_file_alloc_security() in my kernel, and each of those does a cond_resched() via the might_sleep() in the slub code. This test is doing ~400k open/closes per second per CPU, so that's ~1.2M cond_resched()/sec/CPU, but that's still hundreds of ns between calls on average.
I'll do some more ftraces and dig in to those debugfs files early next week.
Quoted 3 lines
But please feel free to send along your patch, CCing LKML. Longer term, I probably need to take a more algorithmic approach, but what you have will be useful to benchmarkers until then.
With the caveat that I exerted approximately 15 seconds of brainpower to code it up...patch attached.
---
b/arch/x86/kernel/nmi.c | 3 +++ b/include/linux/rcupdate.h | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~dirty-rcu-hack +3 -0
@@ -88,10 +88,13 @@ __setup("unknown_nmi_panic", setup_unknostatic u64 nmi_longest_ns =1 *NSEC_PER_MSEC ; +u64 RCU_COND_RESCHED_LIM =256 ;static int __init nmi_warning_debugfs (void ) {debugfs_create_u64 ("nmi_longest_ns" ,0644 , arch_debugfs_dir, &nmi_longest_ns); +debugfs_create_u64 ("RCU_COND_RESCHED_LIM" ,0644 , + arch_debugfs_dir, &RCU_COND_RESCHED_LIM );return 0 ; }fs_initcall (nmi_warning_debugfs);
include/linux/rcupdate.h renamed from include/linux/rcupdate.h~dirty-rcu-hack +1 -1
@@ -303,7 +303,7 @@ bool __rcu_is_watching(void);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -#define RCU_COND_RESCHED_LIM 256/* ms vs. 100s of ms. */ +extern u64 RCU_COND_RESCHED_LIM /* ms vs. 100s of ms. */DECLARE_PER_CPU (int, rcu_cond_resched_count);void rcu_resched (void );
_
Quoted 32 lines · Dave Hansen
Quoted 16 lines · Paul E. McKenney
On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dav
Quoted 6 lines
So, I bisected it down to this:
Quoted 4 lines
commit
ac1bea85781e9004da9b3e8a4b097c18492d857cAuthor: Paul E. McKenney <[email protected]> Date: Sun Mar 16 21:36:25 2014 -0700Quoted 8 lines
Quoted 1 line
sched,rcu: Make
cond_resched()report RCU quiescent statesSpecifically, if I raise
RCU_COND_RESCHED_LIM, things get back to their 3.15 levels.Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quoted 7 lines
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is
rcu_preemptforCONFIG_PREEMPTandrcu_schedfor !CONFIG_PREEMPT.Another possibility is that someone is invoking
cond_reched()in an incredibly tight loop.
open()does at least a couple of allocations ingetname(),get_empty_filp()andapparmor_file_alloc_security()in my kernel, and each of those does acond_resched()via themight_sleep()in the slub code. This test is doing ~400k open/closes per second per CPU, so that's ~1.2M cond_resched()/sec/CPU, but that's still hundreds of ns between calls on average.
Quoted 8 lines
I'll do some more ftraces and dig in to those debugfs files early next week.
Quoted 3 lines
But please feel free to send along your patch, CCing LKML. Longer term, I probably need to take a more algorithmic approach, but what you have will be useful to benchmarkers until then.
With the caveat that I exerted approximately 15 seconds of brainpower to code it up...patch attached.
Thank you Dave! And if someone doesn't like it, they can always improve upon it, right? ;-)
Thanx, Paul
Quoted 36 lines
---
b/arch/x86/kernel/nmi.c | 3 +++ b/include/linux/rcupdate.h | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)diff -puN
arch/x86/kernel/nmi.c~dirty-rcu-hackarch/x86/kernel/nmi.carch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~dirty-rcu-hack +3
-0@@ -88,10 +88,13 @@ __setup("unknown_nmi_panic", setup_unknostatic u64 nmi_longest_ns =1 *NSEC_PER_MSEC ; +u64 RCU_COND_RESCHED_LIM =256 ;static int __init nmi_warning_debugfs (void ) {debugfs_create_u64 ("nmi_longest_ns" ,0644 , arch_debugfs_dir, &nmi_longest_ns); +debugfs_create_u64 ("RCU_COND_RESCHED_LIM" ,0644 , + arch_debugfs_dir, &RCU_COND_RESCHED_LIM );return 0 ; }fs_initcall (nmi_warning_debugfs);diff -puN
include/linux/rcupdate.h~dirty-rcu-hackinclude/linux/rcupdate.hinclude/linux/rcupdate.h renamed from include/linux/rcupdate.h~dirty-rcu-hack +1
-1@@ -303,7 +303,7 @@ bool __rcu_is_watching(void);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -#define RCU_COND_RESCHED_LIM 256/* ms vs. 100s of ms. */ +extern u64 RCU_COND_RESCHED_LIM /* ms vs. 100s of ms. */DECLARE_PER_CPU (int, rcu_cond_resched_count);void rcu_resched (void );_
Quoted 7 lines · Paul E. McKenney
Quoted 3 lines
Quoted 3 lines
Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is
rcu_preemptforCONFIG_PREEMPTandrcu_schedfor !CONFIG_PREEMPT.
With the previously-mentioned workload, rcugp's "age" averages 9 with the old kernel (or RCU_COND_RESCHED_LIM at a high value) and 2 with the current kernel which contains this regression.
I also checked the rate and sources for how I'm calling cond_resched. I'm calling it 5x for every open/close() pair in my test case, which take about 7us. So, _cond_resched() is, on average, only being called every microsecond. That doesn't seem _too_ horribly extreme.
Quoted 11 lines
3895.165846 | 8) | SyS_open() {
3895.165846 | 8) 0.065 us | _cond_resched();
3895.165847 | 8) 0.064 us | _cond_resched();
3895.165849 | 8) 2.406 us | }
3895.165849 | 8) 0.199 us | SyS_close();
3895.165850 | 8) | do_notify_resume() {
3895.165850 | 8) 0.063 us | _cond_resched();
3895.165851 | 8) 0.069 us | _cond_resched();
3895.165852 | 8) 0.060 us | _cond_resched();
3895.165852 | 8) 2.194 us | }
3895.165853 | 8) | SyS_open() {The more I think about it, the more I think we can improve on a purely call-based counter.
First, it couples the number of cond_resched() directly calls with the benefits we see out of RCU. We really don't *need* to see more grace periods if we have more cond_resched() calls.
It also ends up eating a new cacheline in a bunch of pretty hot paths. It would be nice to be able to keep the fast path part of this as at least read-only.
Could we do something (functionally) like the attached patch? Instead of counting cond_resched() calls, we could just specify some future time by which we want have a quiescent state. We could even push the time to be something _just_ before we would have declared a stall.
---
b/arch/x86/kernel/nmi.c | 6 +++--- b/include/linux/rcupdate.h | 7 +++---- b/kernel/rcu/update.c | 4 ++-- 3 files changed, 8 insertions(+), 9 deletions(-)
include/linux/rcupdate.h renamed from include/linux/rcupdate.h~rcu-halfstall +3 -4
@@ -303,8 +303,8 @@ bool __rcu_is_watching(void);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -extern u64 RCU_COND_RESCHED_LIM ;/* ms vs. 100s of ms. */ -DECLARE_PER_CPU (int, rcu_cond_resched_count); +extern u64 RCU_COND_RESCHED_EVERY_THIS_JIFFIES ; +DECLARE_PER_CPU(unsigned long, rcu_cond_resched_at_jiffies);void rcu_resched (void );/*
@@ -321,8 +321,7 @@ void rcu_resched(void);*/ static inline bool rcu_should_resched (void ) { -return raw_cpu_inc_return (rcu_cond_resched_count) >= -RCU_COND_RESCHED_LIM ; +return raw_cpu_read (rcu_cond_resched_at_jiffies) >= jiffies; }/*
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~rcu-halfstall +3 -3
@@ -88,13 +88,13 @@ __setup("unknown_nmi_panic", setup_unknostatic u64 nmi_longest_ns =1 *NSEC_PER_MSEC ; -u64 RCU_COND_RESCHED_LIM =256 ; +u64 RCU_COND_RESCHED_EVERY_THIS_JIFFIES =100 ;static int __init nmi_warning_debugfs (void ) {debugfs_create_u64 ("nmi_longest_ns" ,0644 , arch_debugfs_dir, &nmi_longest_ns); -debugfs_create_u64 ("RCU_COND_RESCHED_LIM" ,0644 , - arch_debugfs_dir, &RCU_COND_RESCHED_LIM ); +debugfs_create_u64 ("RCU_COND_RESCHED_EVERY_THIS_JIFFIES" ,0644 , + arch_debugfs_dir, &RCU_COND_RESCHED_EVERY_THIS_JIFFIES );return 0 ; }fs_initcall (nmi_warning_debugfs);
kernel/rcu/update.c renamed from kernel/rcu/update.c~rcu-halfstall +2 -2
@@ -355,7 +355,7 @@ early_initcall(check_cpu_stall_init);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -DEFINE_PER_CPU (int, rcu_cond_resched_count); +DEFINE_PER_CPU(unsigned long, rcu_cond_resched_at_jiffies);/* * Report a set of RCU quiescent states, for use by cond_resched()
@@ -364,7 +364,7 @@ DEFINE_PER_CPU(int, rcu_cond_resched_couvoid rcu_resched (void ) {preempt_disable (); -__this_cpu_write (rcu_cond_resched_count,0 ); +__this_cpu_write (rcu_cond_resched_at_jiffies, jiffies +RCU_COND_RESCHED_EVERY_THIS_JIFFIES );rcu_note_context_switch (smp_processor_id ());preempt_enable (); }
_
Josh Triplett
maintainerQuoted 37 lines · Dave Hansen
Quoted 7 lines · Paul E. McKenney
Quoted 3 lines
Quoted 3 lines
Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is
rcu_preemptforCONFIG_PREEMPTandrcu_schedfor !CONFIG_PREEMPT.With the previously-mentioned workload, rcugp's "age" averages 9 with the old kernel (or
RCU_COND_RESCHED_LIMat a high value) and 2 with the current kernel which contains this regression.I also checked the rate and sources for how I'm calling
cond_resched. I'm calling it 5x for every open/close() pair in my test case, which take about 7us. So,_cond_resched()is, on average, only being called every microsecond. That doesn't seem _too_ horribly extreme.Quoted 11 lines
3895.165846 | 8) | SyS_open() { 3895.165846 | 8) 0.065 us | _cond_resched(); 3895.165847 | 8) 0.064 us | _cond_resched(); 3895.165849 | 8) 2.406 us | } 3895.165849 | 8) 0.199 us | SyS_close(); 3895.165850 | 8) | do_notify_resume() { 3895.165850 | 8) 0.063 us | _cond_resched(); 3895.165851 | 8) 0.069 us | _cond_resched(); 3895.165852 | 8) 0.060 us | _cond_resched(); 3895.165852 | 8) 2.194 us | } 3895.165853 | 8) | SyS_open() {The more I think about it, the more I think we can improve on a purely call-based counter.
First, it couples the number of
cond_resched()directly calls with the benefits we see out of RCU. We really don't *need* to see more grace periods if we have morecond_resched()calls.
Quoted 8 lines
It also ends up eating a new cacheline in a bunch of pretty hot paths. It would be nice to be able to keep the fast path part of this as at least read-only.
Could we do something (functionally) like the attached patch? Instead of counting
cond_resched()calls, we could just specify some future time by which we want have a quiescent state. We could even push the time to be something _just_ before we would have declared a stall.
Looks quite promising to me, as long as the CPU in question is actively updating jiffies. I'd love to see some numbers from that approach.
- Josh Triplett
Quoted 8 lines
It also ends up eating a new cacheline in a bunch of pretty hot paths. It would be nice to be able to keep the fast path part of this as at least read-only.
Could we do something (functionally) like the attached patch? Instead of counting
cond_resched()calls, we could just specify some future time by which we want have a quiescent state. We could even push the time to be something _just_ before we would have declared a stall.
I still think it's totally the wrong place. cond_resched() is in so many fast paths (every lock, every allocation). It just doesn't make sense to add non essential things like this to it.
I would be rather to just revert the original patch.
-Andi
Quoted 37 lines · Dave Hansen
Quoted 7 lines · Paul E. McKenney
Quoted 3 lines
Quoted 3 lines
Could the additional RCU quiescent states be causing us to be doing more RCU frees that we were before, and getting less benefit from the lock batching that RCU normally provides?
Quite possibly. One way to check would be to use the debugfs files rcu/*/rcugp, which give a count of grace periods since boot for each RCU flavor. Here "*" is
rcu_preemptforCONFIG_PREEMPTandrcu_schedfor !CONFIG_PREEMPT.With the previously-mentioned workload, rcugp's "age" averages 9 with the old kernel (or
RCU_COND_RESCHED_LIMat a high value) and 2 with the current kernel which contains this regression.I also checked the rate and sources for how I'm calling
cond_resched. I'm calling it 5x for every open/close() pair in my test case, which take about 7us. So,_cond_resched()is, on average, only being called every microsecond. That doesn't seem _too_ horribly extreme.Quoted 11 lines
3895.165846 | 8) | SyS_open() { 3895.165846 | 8) 0.065 us | _cond_resched(); 3895.165847 | 8) 0.064 us | _cond_resched(); 3895.165849 | 8) 2.406 us | } 3895.165849 | 8) 0.199 us | SyS_close(); 3895.165850 | 8) | do_notify_resume() { 3895.165850 | 8) 0.063 us | _cond_resched(); 3895.165851 | 8) 0.069 us | _cond_resched(); 3895.165852 | 8) 0.060 us | _cond_resched(); 3895.165852 | 8) 2.194 us | } 3895.165853 | 8) | SyS_open() {The more I think about it, the more I think we can improve on a purely call-based counter.
First, it couples the number of
cond_resched()directly calls with the benefits we see out of RCU. We really don't *need* to see more grace periods if we have morecond_resched()calls.
Quoted 8 lines
It also ends up eating a new cacheline in a bunch of pretty hot paths. It would be nice to be able to keep the fast path part of this as at least read-only.
Could we do something (functionally) like the attached patch? Instead of counting
cond_resched()calls, we could just specify some future time by which we want have a quiescent state. We could even push the time to be something _just_ before we would have declared a stall.
Nice analysis!
So if I understand correctly, a goodly part of the regression is due not to the overhead added to cond_resched(), but rather because grace periods are now happening faster, thus incurring more overhead. Is that correct?
If this is the case, could you please let me know roughly how sensitive is the performance to the time delay in RCU_COND_RESCHED_EVERY_THIS_JIFFIES?
The patch looks promising. I will probably drive the time-setup deeper into the guts of RCU, which should allow moving the access to jiffies and the comparison off of the fast path as well, but this appears to me to be good and sufficient for others encountering this same problem in the meantime.
And of course, if my attempt to drive things deeper into the guts of RCU doesn't work out, something close to the below might also be the eventual solution.
Thanx, Paul
Quoted 73 lines
---
b/arch/x86/kernel/nmi.c | 6 +++--- b/include/linux/rcupdate.h | 7 +++---- b/kernel/rcu/update.c | 4 ++-- 3 files changed, 8 insertions(+), 9 deletions(-)diff -puN
include/linux/rcupdate.h~rcu-halfstallinclude/linux/rcupdate.hinclude/linux/rcupdate.h renamed from include/linux/rcupdate.h~rcu-halfstall +3
-4@@ -303,8 +303,8 @@ bool __rcu_is_watching(void);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -extern u64 RCU_COND_RESCHED_LIM ;/* ms vs. 100s of ms. */ -DECLARE_PER_CPU (int, rcu_cond_resched_count); +extern u64 RCU_COND_RESCHED_EVERY_THIS_JIFFIES ; +DECLARE_PER_CPU(unsigned long, rcu_cond_resched_at_jiffies);void rcu_resched (void );/* @@ -321,8 +321,7 @@ void rcu_resched(void);*/ static inline bool rcu_should_resched (void ) { -return raw_cpu_inc_return (rcu_cond_resched_count) >= -RCU_COND_RESCHED_LIM ; +return raw_cpu_read (rcu_cond_resched_at_jiffies) >= jiffies; }/* diff -puN
arch/x86/kernel/nmi.c~rcu-halfstallarch/x86/kernel/nmi.carch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~rcu-halfstall +3
-3@@ -88,13 +88,13 @@ __setup("unknown_nmi_panic", setup_unknostatic u64 nmi_longest_ns =1 *NSEC_PER_MSEC ; -u64 RCU_COND_RESCHED_LIM =256 ; +u64 RCU_COND_RESCHED_EVERY_THIS_JIFFIES =100 ;static int __init nmi_warning_debugfs (void ) {debugfs_create_u64 ("nmi_longest_ns" ,0644 , arch_debugfs_dir, &nmi_longest_ns); -debugfs_create_u64 ("RCU_COND_RESCHED_LIM" ,0644 , - arch_debugfs_dir, &RCU_COND_RESCHED_LIM ); +debugfs_create_u64 ("RCU_COND_RESCHED_EVERY_THIS_JIFFIES" ,0644 , + arch_debugfs_dir, &RCU_COND_RESCHED_EVERY_THIS_JIFFIES );return 0 ; }fs_initcall (nmi_warning_debugfs);diff -puN
kernel/rcu/update.c~rcu-halfstallkernel/rcu/update.ckernel/rcu/update.c renamed from kernel/rcu/update.c~rcu-halfstall +2
-2@@ -355,7 +355,7 @@ early_initcall(check_cpu_stall_init);* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. */ -DEFINE_PER_CPU (int, rcu_cond_resched_count); +DEFINE_PER_CPU(unsigned long, rcu_cond_resched_at_jiffies);/* * Report a set of RCU quiescent states, for use by cond_resched() @@ -364,7 +364,7 @@ DEFINE_PER_CPU(int, rcu_cond_resched_couvoid rcu_resched (void ) {preempt_disable (); -__this_cpu_write (rcu_cond_resched_count,0 ); +__this_cpu_write (rcu_cond_resched_at_jiffies, jiffies +RCU_COND_RESCHED_EVERY_THIS_JIFFIES );rcu_note_context_switch (smp_processor_id ());preempt_enable (); }_
Quoted 9 lines · Andi Kleen
Quoted 8 lines
It also ends up eating a new cacheline in a bunch of pretty hot paths. It would be nice to be able to keep the fast path part of this as at least read-only.
Could we do something (functionally) like the attached patch? Instead of counting
cond_resched()calls, we could just specify some future time by which we want have a quiescent state. We could even push the time to be something _just_ before we would have declared a stall.
Quoted 5 lines
I still think it's totally the wrong place.
cond_resched()is in so many fast paths (every lock, every allocation). It just doesn't make sense to add non essential things like this to it.I would be rather to just revert the original patch.
OK. What would you suggest instead? If all we do is to revert the original patch, we once again end up with long-running in-kernel code paths stalling the RCU grace period. The cond_resched() calls sprinkled through them once again won't help with this.
Or are you suggesting leveraging the now-deprecated set_need_resched() so that the checks happen deeper in the scheduler? Looks like grabbing the offending CPU's task and doing set_tsk_need_resched() on that task is the replacement.
CCing Peter Zijlstra for his thoughts on this.
Thanx, Paul
Quoted 1 line
OK. What would you suggest instead? If all we do is to revert the
Hang checker should have two timer phases:
Timer fires first time:
- Save context switch counter on that. Force a reschedule to some work queue. Rearm timer
Timer fires again:
- Check reschedule count. If the reschedule count changed it was a real hang, otherwise ignore.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 1 line
OK. What would you suggest instead? If all we do is to revert the
Hang checker should have two timer phases:
Timer fires first time:
- Save context switch counter on that. Force a reschedule to some work queue. Rearm timerTimer fires again:
- Check reschedule count. If the reschedule count changed it was a real hang, otherwise ignore.
I could take that approach, but the RT guys aren't going to thank me for the wakeup associated with the work queue. I suppose that I could use one approach for real-time workloads and your workqueue-base approach for other workloads.
Still, I bet I can drop the common-case cond_resched() overhead to a single read of a per-CPU variable and a branch. But yes, that would be in response to the second phase, FWIW. If that measurably too much overhead, then one thing for realtime and another otherwise.
Thanx, Paul
Quoted 3 lines · Paul E. McKenney
So if I understand correctly, a goodly part of the regression is due not to the overhead added to
cond_resched(), but rather because grace periods are now happening faster, thus incurring more overhead. Is that correct?
Yes, that's the theory at least.
Quoted 2 lines
If this is the case, could you please let me know roughly how sensitive is the performance to the time delay in
RCU_COND_RESCHED_EVERY_THIS_JIFFIES?
This is the previous kernel, plus RCU tracing, so it's not 100% apples-to-apples (and it peaks a bit lower than the other kernel). But here's the will-it-scale open1 throughput on the y axis vs RCU_COND_RESCHED_EVERY_THIS_JIFFIES on x:
http://sr71.net/~dave/intel/jiffies-vs-openops.png
This was a quick and dirty single run with very little averaging, so I expect there to be a good amount of noise. I ran it from 1->100, but it seemed to peak at about 30.
Quoted 5 lines
The patch looks promising. I will probably drive the time-setup deeper into the guts of RCU, which should allow moving the access to jiffies and the comparison off of the fast path as well, but this appears to me to be good and sufficient for others encountering this same problem in the meantime.
Yeah, the more overhead we can push out of cond_resched(), the better. I had no idea how much we call it!
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Andi Kleen
Quoted 1 line
OK. What would you suggest instead? If all we do is to revert the
Hang checker should have two timer phases:
Timer fires first time:
- Save context switch counter on that. Force a reschedule to some work queue. Rearm timer
Quoted 6 lines
Quoted 3 lines
Timer fires again:
- Check reschedule count. If the reschedule count changed it was a real hang, otherwise ignore.I could take that approach, but the RT guys aren't going to thank me for the wakeup associated with the work queue. I suppose that I could use
They can disable the hang timer if it's really problem.
If they cannot tolerate a single context switch they likely cannot tolerate a timer firing either.
-Andi
Quoted 14 lines · Andi Kleen
Quoted 13 lines · Paul E. McKenney
Quoted 11 lines · Andi Kleen
Quoted 1 line
OK. What would you suggest instead? If all we do is to revert the
Hang checker should have two timer phases:
Timer fires first time:
- Save context switch counter on that. Force a reschedule to some work queue. Rearm timerTimer fires again:
- Check reschedule count. If the reschedule count changed it was a real hang, otherwise ignore.
Quoted 7 lines
Quoted 2 lines
I could take that approach, but the RT guys aren't going to thank me for the wakeup associated with the work queue. I suppose that I could use
They can disable the hang timer if it's really problem.
If they cannot tolerate a single context switch they likely cannot tolerate a timer firing either.
Ah, but I make the timer fire on some other CPU.
Thanx, Paul
Quoted 15 lines · Dave Hansen
Quoted 3 lines · Paul E. McKenney
So if I understand correctly, a goodly part of the regression is due not to the overhead added to
cond_resched(), but rather because grace periods are now happening faster, thus incurring more overhead. Is that correct?Yes, that's the theory at least.
Quoted 2 lines
If this is the case, could you please let me know roughly how sensitive is the performance to the time delay in
RCU_COND_RESCHED_EVERY_THIS_JIFFIES?This is the previous kernel, plus RCU tracing, so it's not 100% apples-to-apples (and it peaks a bit lower than the other kernel). But here's the will-it-scale open1 throughput on the y axis vs
RCU_COND_RESCHED_EVERY_THIS_JIFFIESon x:
Quoted 5 lines
http://sr71.net/~dave/intel/jiffies-vs-openops.pngThis was a quick and dirty single run with very little averaging, so I expect there to be a good amount of noise. I ran it from 1->100, but it seemed to peak at about 30.
OK, so a default setting on the order of 20-30 jiffies looks promising.
Quoted 8 lines
Quoted 5 lines
The patch looks promising. I will probably drive the time-setup deeper into the guts of RCU, which should allow moving the access to jiffies and the comparison off of the fast path as well, but this appears to me to be good and sufficient for others encountering this same problem in the meantime.
Yeah, the more overhead we can push out of
cond_resched(), the better. I had no idea how much we call it!
Me neither!
Thanx, Paul
Quoted 5 lines · Andi Kleen
I still think it's totally the wrong place.
cond_resched()is in so many fast paths (every lock, every allocation). It just doesn't make sense to add non essential things like this to it.I would be rather to just revert the original patch.
I fully agree.
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines
Quoted 10 lines
This is the previous kernel, plus RCU tracing, so it's not 100% apples-to-apples (and it peaks a bit lower than the other kernel). But here's the will-it-scale open1 throughput on the y axis vs
RCU_COND_RESCHED_EVERY_THIS_JIFFIESon x:http://sr71.net/~dave/intel/jiffies-vs-openops.pngThis was a quick and dirty single run with very little averaging, so I expect there to be a good amount of noise. I ran it from 1->100, but it seemed to peak at about 30.
OK, so a default setting on the order of 20-30 jiffies looks promising.
For the biggest machine I have today, yeah. But, we need to be a bit careful here. The CPUs I'm running it on were released 3 years ago and I think we need to be planning at _least_ for today's large systems. I would guess that by raising ...EVERY_THIS_JIFFIES, we're shifting this curve out to the right:
http://sr71.net/~dave/intel/3.16-open1regression-0.png
so that we're _just_ before the regression hits us. But that just guarantees I'll hit this again when I get new CPUs. :)
If we go this route, I think we should probably take it up in to the 100-200 range, or even scale it to something on the order of what the rcu stall timeout is. Other than the stall detector, is there some other reason to be forcing frequent quiescent states?
Quoted 19 lines · Dave Hansen
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines
Quoted 10 lines
This is the previous kernel, plus RCU tracing, so it's not 100% apples-to-apples (and it peaks a bit lower than the other kernel). But here's the will-it-scale open1 throughput on the y axis vs
RCU_COND_RESCHED_EVERY_THIS_JIFFIESon x:http://sr71.net/~dave/intel/jiffies-vs-openops.pngThis was a quick and dirty single run with very little averaging, so I expect there to be a good amount of noise. I ran it from 1->100, but it seemed to peak at about 30.
OK, so a default setting on the order of 20-30 jiffies looks promising.
For the biggest machine I have today, yeah. But, we need to be a bit careful here. The CPUs I'm running it on were released 3 years ago and I think we need to be planning at _least_ for today's large systems. I would guess that by raising ...EVERY_THIS_JIFFIES, we're shifting this curve out to the right:
Quoted 4 lines
http://sr71.net/~dave/intel/3.16-open1regression-0.pngso that we're _just_ before the regression hits us. But that just guarantees I'll hit this again when I get new CPUs. :)
Understood. One approach would be to scale this in a manner similar to the scaling of the delay from the beginning of the grace period to the start of quiescent-state forcing, which is about three jiffies on small systems scaling up to about 20 jiffies on large systems.
Quoted 4 lines
If we go this route, I think we should probably take it up in to the 100-200 range, or even scale it to something on the order of what the rcu stall timeout is. Other than the stall detector, is there some other reason to be forcing frequent quiescent states?
Yep. CONFIG_NO_HZ_FULL+nohz_full kernels running in kernel mode don't progress RCU grace periods. But they should not need to be all that frequent.
Thanx, Paul
Quoted 11 lines · Dave Hansen
Hi Paul,
I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop:
At about 50 cores worth of processes, 3.15 and the pre-3.16 code start to diverge, with 3.15 scaling better:
Quoted 7 lines
http://sr71.net/~dave/intel/3.16-open1regression-0.pngSome profiles point to a big increase in contention inside slub.c's
get_partial_node()(the allocation side of the slub code) causing the regression. That particularopen()test is known to do a lot of slab operations. But, the odd part is that the slub code hasn't been touched much.
Coming back to this... If the original was stalling RCU grace periods for the duration of the test, then it would also be deferring any freeing until after the end of the test. This is of course similar to the usual Java benchmarking trick of making sure that the garbage collector never runs. It would also mean that if a change caused RCU grace periods to complete during the test, that change might appear to reduce throughput when in fact it was simply causing the throughput to be more accurately represented.
The reason I bring this possibility up is that it would account for the increase in contention in slub -- by causing free operations to occur concurrently.
Thanx, Paul
Quoted 20 lines · Paul E. McKenney
Quoted 11 lines · Dave Hansen
Hi Paul,
I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop:
At about 50 cores worth of processes, 3.15 and the pre-3.16 code start to diverge, with 3.15 scaling better:
Quoted 7 lines
http://sr71.net/~dave/intel/3.16-open1regression-0.pngSome profiles point to a big increase in contention inside slub.c's
get_partial_node()(the allocation side of the slub code) causing the regression. That particularopen()test is known to do a lot of slab operations. But, the odd part is that the slub code hasn't been touched much.
Quoted 3 lines
Coming back to this... If the original was stalling RCU grace periods for the duration of the test, then it would also be deferring any freeing until after the end of the test.
I run the test for pretty long periods of time, and I don't see any consistent growth in the memory used. I'd expect if we were fully stalling RCU grace periods that we'd see memory usage grow too.
Looking at rcu_sched/rcugp at 1-second intervals, we can see the gpnum going up consistently (incrementing at a rate of about 15/second).
Does that rule out the possibility of stalling them until after the test?
Quoted 22 lines
Wed Jun 18 14:46:32 PDT 2014: completed=7707 gpnum=7708 age=14 max=123, 7 Wed Jun 18 14:46:33 PDT 2014: completed=7723 gpnum=7724 age=14 max=123, 7 Wed Jun 18 14:46:34 PDT 2014: completed=7739 gpnum=7740 age=19 max=123, 7 Wed Jun 18 14:46:35 PDT 2014: completed=7756 gpnum=7757 age=3 max=123, 6 Wed Jun 18 14:46:36 PDT 2014: completed=7771 gpnum=7772 age=12 max=123, 5 Wed Jun 18 14:46:37 PDT 2014: completed=7787 gpnum=7788 age=7 max=123, 5 Wed Jun 18 14:46:39 PDT 2014: completed=7802 gpnum=7803 age=14 max=123, 5 Wed Jun 18 14:46:40 PDT 2014: completed=7816 gpnum=7817 age=15 max=123, 5 Wed Jun 18 14:46:41 PDT 2014: completed=7831 gpnum=7832 age=15 max=123, 6 Wed Jun 18 14:46:42 PDT 2014: completed=7847 gpnum=7848 age=15 max=123, 6 Wed Jun 18 14:46:43 PDT 2014: completed=7860 gpnum=7861 age=15 max=123, 6 Wed Jun 18 14:46:44 PDT 2014: completed=7877 gpnum=7878 age=15 max=123, 6 Wed Jun 18 14:46:45 PDT 2014: completed=7892 gpnum=7893 age=17 max=123, 6 Wed Jun 18 14:46:46 PDT 2014: completed=7906 gpnum=7907 age=17 max=123, 7 Wed Jun 18 14:46:47 PDT 2014: completed=7920 gpnum=7921 age=19 max=123, 7 Wed Jun 18 14:46:48 PDT 2014: completed=7936 gpnum=7937 age=19 max=123, 7 Wed Jun 18 14:46:49 PDT 2014: completed=7954 gpnum=7955 age=2 max=123, 7 Wed Jun 18 14:46:50 PDT 2014: completed=7970 gpnum=7971 age=0 max=123, 7 Wed Jun 18 14:46:51 PDT 2014: completed=7985 gpnum=7986 age=6 max=123, 7 Wed Jun 18 14:46:52 PDT 2014: completed=8000 gpnum=8001 age=7 max=123, 8 Wed Jun 18 14:46:53 PDT 2014: completed=8015 gpnum=8016 age=8 max=123, 8 Wed Jun 18 14:46:54 PDT 2014: completed=8031 gpnum=8032 age=10 max=123, 7
Quoted 25 lines · Dave Hansen
Quoted 20 lines · Paul E. McKenney
Quoted 11 lines · Dave Hansen
Hi Paul,
I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop:
At about 50 cores worth of processes, 3.15 and the pre-3.16 code start to diverge, with 3.15 scaling better:
Quoted 7 lines
http://sr71.net/~dave/intel/3.16-open1regression-0.pngSome profiles point to a big increase in contention inside slub.c's
get_partial_node()(the allocation side of the slub code) causing the regression. That particularopen()test is known to do a lot of slab operations. But, the odd part is that the slub code hasn't been touched much.Quoted 3 lines
Coming back to this... If the original was stalling RCU grace periods for the duration of the test, then it would also be deferring any freeing until after the end of the test.
Quoted 8 lines
I run the test for pretty long periods of time, and I don't see any consistent growth in the memory used. I'd expect if we were fully stalling RCU grace periods that we'd see memory usage grow too.
Looking at rcu_sched/rcugp at 1-second intervals, we can see the gpnum going up consistently (incrementing at a rate of about 15/second).
Does that rule out the possibility of stalling them until after the test?
That it does!
Quoted 14 lines
Quoted 14 lines
Wed Jun 18 14:46:32 PDT 2014: completed=7707 gpnum=7708 age=14 max=123, 7 Wed Jun 18 14:46:33 PDT 2014: completed=7723 gpnum=7724 age=14 max=123, 7 Wed Jun 18 14:46:34 PDT 2014: completed=7739 gpnum=7740 age=19 max=123, 7 Wed Jun 18 14:46:35 PDT 2014: completed=7756 gpnum=7757 age=3 max=123, 6 Wed Jun 18 14:46:36 PDT 2014: completed=7771 gpnum=7772 age=12 max=123, 5 Wed Jun 18 14:46:37 PDT 2014: completed=7787 gpnum=7788 age=7 max=123, 5 Wed Jun 18 14:46:39 PDT 2014: completed=7802 gpnum=7803 age=14 max=123, 5 Wed Jun 18 14:46:40 PDT 2014: completed=7816 gpnum=7817 age=15 max=123, 5 Wed Jun 18 14:46:41 PDT 2014: completed=7831 gpnum=7832 age=15 max=123, 6 Wed Jun 18 14:46:42 PDT 2014: completed=7847 gpnum=7848 age=15 max=123, 6 Wed Jun 18 14:46:43 PDT 2014: completed=7860 gpnum=7861 age=15 max=123, 6 Wed Jun 18 14:46:44 PDT 2014: completed=7877 gpnum=7878 age=15 max=123, 6 Wed Jun 18 14:46:45 PDT 2014: completed=7892 gpnum=7893 age=17 max=123, 6 Wed Jun 18 14:46:46 PDT 2014: completed=7906 gpnum=7907 age=17 max=123, 7
Quoted 8 lines
Quoted 8 lines
Wed Jun 18 14:46:47 PDT 2014: completed=7920 gpnum=7921 age=19 max=123, 7 Wed Jun 18 14:46:48 PDT 2014: completed=7936 gpnum=7937 age=19 max=123, 7 Wed Jun 18 14:46:49 PDT 2014: completed=7954 gpnum=7955 age=2 max=123, 7 Wed Jun 18 14:46:50 PDT 2014: completed=7970 gpnum=7971 age=0 max=123, 7 Wed Jun 18 14:46:51 PDT 2014: completed=7985 gpnum=7986 age=6 max=123, 7 Wed Jun 18 14:46:52 PDT 2014: completed=8000 gpnum=8001 age=7 max=123, 8 Wed Jun 18 14:46:53 PDT 2014: completed=8015 gpnum=8016 age=8 max=123, 8 Wed Jun 18 14:46:54 PDT 2014: completed=8031 gpnum=8032 age=10 max=123, 7
Thank you for checking!
Thanx, Paul
Quoted 30 lines · Paul E. McKenney
Quoted 19 lines · Dave Hansen
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines
Quoted 10 lines
This is the previous kernel, plus RCU tracing, so it's not 100% apples-to-apples (and it peaks a bit lower than the other kernel). But here's the will-it-scale open1 throughput on the y axis vs
RCU_COND_RESCHED_EVERY_THIS_JIFFIESon x:http://sr71.net/~dave/intel/jiffies-vs-openops.pngThis was a quick and dirty single run with very little averaging, so I expect there to be a good amount of noise. I ran it from 1->100, but it seemed to peak at about 30.
OK, so a default setting on the order of 20-30 jiffies looks promising.
For the biggest machine I have today, yeah. But, we need to be a bit careful here. The CPUs I'm running it on were released 3 years ago and I think we need to be planning at _least_ for today's large systems. I would guess that by raising ...EVERY_THIS_JIFFIES, we're shifting this curve out to the right:
Quoted 4 lines
http://sr71.net/~dave/intel/3.16-open1regression-0.pngso that we're _just_ before the regression hits us. But that just guarantees I'll hit this again when I get new CPUs. :)
Understood. One approach would be to scale this in a manner similar to the scaling of the delay from the beginning of the grace period to the start of quiescent-state forcing, which is about three jiffies on small systems scaling up to about 20 jiffies on large systems.
Quoted 8 lines
Quoted 4 lines
If we go this route, I think we should probably take it up in to the 100-200 range, or even scale it to something on the order of what the rcu stall timeout is. Other than the stall detector, is there some other reason to be forcing frequent quiescent states?
Yep. CONFIG_NO_HZ_FULL+nohz_full kernels running in kernel mode don't progress RCU grace periods. But they should not need to be all that frequent.
Here is an early version of a patch, which looks promising on short rcutorture tests. It does not yet have control of the holdoff time (I intend to add a module parameter for this), but it also avoids ever having cond_resched() do a quiescent state if there is no grace period in progress or if the current grace period is less than seven jiffies old. (The constant "7" is the thing that will be made into a module parameter.) These restrictions lead me to believe that "7" will perform well in your tests, because normal workloads would almost never have cond_resched() do anything other than a test of a per-CPU variable. But of course your tests are the final judges of that.
Thoughts?
Thanx, Paul
------------------------------------------------------------------------
rcu: Reduce overhead of cond_resched() checks for RCU
Commit ac1bea85781e (Make cond_resched() report RCU quiescent states) fixed a problem where a CPU looping in the kernel with but one runnable task would give RCU CPU stall warnings, even if the in-kernel loop contained cond_resched() calls. Unfortunately, in so doing, it introduced performance regressions in Anton Blanchard's will-it-scale "open1" test. The problem appears to be not so much the increased cond_resched() path length as an increase in the rate at which grace periods complete, which increased per-update grace-period overhead.
This commit takes a different approach to fixing this bug, mainly by avoiding having cond_resched() do an RCU-visible quiescent state unless there is a grace period that has been in flight for a significant period of time. This commit also reduces the common-case cond_resched() overhead to a check of a single per-CPU variable.
include/linux/rcupdate.h +0 -35
@@ -300,41 +300,6 @@ bool __rcu_is_watching(void);#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP) */ /* -* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. -*/ - -#define RCU_COND_RESCHED_LIM 256/* ms vs. 100s of ms. */ -DECLARE_PER_CPU (int, rcu_cond_resched_count); -void rcu_resched (void ); - -/* -* Is it time to report RCU quiescent states? -* -* Note unsynchronized access to rcu_cond_resched_count. Yes, we might -* increment some random CPU's count, and possibly also load the result from -* yet another CPU's count. We might even clobber some other CPU's attempt -* to zero its counter. This is all OK because the goal is not precision, -* but rather reasonable amortization of rcu_note_context_switch() overhead -* and extremely high probability of avoiding RCU CPU stall warnings. -* Note that this function has to be preempted in just the wrong place, -* many thousands of times in a row, for anything bad to happen. -*/ -static inline bool rcu_should_resched (void ) -{ -return raw_cpu_inc_return (rcu_cond_resched_count) >= -RCU_COND_RESCHED_LIM ; -} - -/* -* Report quiscent states to RCU if it is time to do so. -*/ -static inline void rcu_cond_resched (void ) -{ -if (unlikely (rcu_should_resched ())) -rcu_resched (); -} - -/* * Infrastructure to implement the synchronize_() primitives in * TREE_RCU and rcu_barrier_() primitives in TINY_RCU. */
include/linux/rcutiny.h +13 -0
@@ -83,6 +83,19 @@ static inline void rcu_note_context_switch(int cpu)rcu_sched_qs (cpu); } +static inline bool rcu_should_resched (void ) +{ +return false; +} + +static inline void rcu_cond_resched (void ) +{ +} + +static inline void rcu_resched (void ) +{ +} +/* * Take advantage of the fact that there is only one CPU, which * allows us to ignore virtualization-based context switches.
include/linux/rcutree.h +16 -0
@@ -46,6 +46,22 @@ static inline void rcu_virt_note_context_switch(int cpu)rcu_note_context_switch (cpu); } +DECLARE_PER_CPU (int, rcu_cond_resched_mask); +void rcu_resched (void ); + +/* Is it time to report RCU quiescent states? */ +static inline bool rcu_should_resched (void ) +{ +return raw_cpu_read (rcu_cond_resched_mask); +} + +/* Report quiescent states to RCU if it is time to do so. */ +static inline void rcu_cond_resched (void ) +{ +if (unlikely (rcu_should_resched ())) +rcu_resched (); +} +void synchronize_rcu_bh (void );void synchronize_sched_expedited (void );void synchronize_rcu_expedited (void );
kernel/rcu/tree.c +69 -6
106 lines not on this page · raw message
kernel/rcu/tree.h +5 -1
@@ -315,6 +315,9 @@ struct rcu_data {/* 4) reasons this CPU needed to be kicked by force_quiescent_state */ unsigned long dynticks_fqs;/* Kicked due to dynticks idle. */ unsigned long offline_fqs;/* Kicked due to being offline. */ +unsigned long cond_resched_completed; +/* Grace period that needs help */ +/* from cond_resched(). */ /* 5) __rcu_pending() statistics. */ unsigned long n_rcu_pending;/* rcu_pending() calls since boot. */
@@ -400,6 +403,7 @@ struct rcu_state {struct rcu_node *level[RCU_NUM_LVLS ];/* Hierarchy levels. */ u32 levelcnt[MAX_RCU_LVLS +1 ];/* # nodes in each level. */ u8 levelspread[RCU_NUM_LVLS ];/* kids/node in each level. */ +u8 flavor_mask;/* bit in flavor mask. */ struct rcu_data __percpu *rda;/* pointer of percu rcu_data. */ void (*call)(struct rcu_head *head,/* call_rcu() flavor. */ void (*func)(struct rcu_head *head));
@@ -571,7 +575,7 @@ static bool rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp);static void do_nocb_deferred_wakeup (struct rcu_data *rdp);static void rcu_boot_init_nocb_percpu_data (struct rcu_data *rdp);static void rcu_spawn_nocb_kthreads (struct rcu_state *rsp); -static void rcu_kick_nohz_cpu (int cpu); +static void __maybe_unused rcu_kick_nohz_cpu (int cpu);static bool init_nocb_callback_list (struct rcu_data *rdp);static void rcu_sysidle_enter (struct rcu_dynticks *rdtp,int irq);static void rcu_sysidle_exit (struct rcu_dynticks *rdtp,int irq);
kernel/rcu/tree_plugin.h +1 -1
@@ -2422,7 +2422,7 @@ static bool init_nocb_callback_list(struct rcu_data *rdp)* if an adaptive-ticks CPU is failing to respond to the current grace * period and has not be idle from an RCU perspective, kick it. */ -static void rcu_kick_nohz_cpu (int cpu) +static void __maybe_unused rcu_kick_nohz_cpu (int cpu) {#ifdef CONFIG_NO_HZ_FULL if (tick_nohz_full_cpu (cpu))
kernel/rcu/update.c +0 -18
@@ -350,21 +350,3 @@ static int __init check_cpu_stall_init(void)early_initcall (check_cpu_stall_init);#endif /* #ifdef CONFIG_RCU_STALL_COMMON */ - -/* -* Hooks for cond_resched() and friends to avoid RCU CPU stall warnings. -*/ - -DEFINE_PER_CPU (int, rcu_cond_resched_count); - -/* -* Report a set of RCU quiescent states, for use by cond_resched() -* and friends. Out of line due to being called infrequently. -*/ -void rcu_resched (void ) -{ -preempt_disable (); -__this_cpu_write (rcu_cond_resched_count,0 ); -rcu_note_context_switch (smp_processor_id ()); -preempt_enable (); -}
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
cond_resched() is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.
If you really wanted to do this I think you would first need to define a cond_resched_i_am_not_fast() or somesuch.
Or put it all behind some debugging ifdef.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
Quoted 7 lines
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.Or put it all behind some debugging ifdef.
My first thought was to put it behind CONFIG_NO_HZ_FULL, but everyone seems to be enabling that one.
As mentioned earlier, I could potentially push the check behind the need-resched check, which would get it off of the common case of the code paths you call out above.
Thanx, Paul
Quoted 14 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
Quoted 6 lines
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.
Quoted 8 lines
Quoted 1 line
Or put it all behind some debugging ifdef.
My first thought was to put it behind
CONFIG_NO_HZ_FULL, but everyone seems to be enabling that one.As mentioned earlier, I could potentially push the check behind the need-resched check, which would get it off of the common case of the code paths you call out above.
Of course, it would also be good to measure this and see how much it really hurts.
Thanx, Paul
Quoted 11 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
Quoted 3 lines
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.
Quoted 7 lines
Quoted 4 lines
If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.Or put it all behind some debugging ifdef.
My first thought was to put it behind
CONFIG_NO_HZ_FULL, but everyone seems to be enabling that one.
Not everybody, SUSE doesn't even have it enabled in factory.
-Mike
Quoted 7 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 7 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
On the bizarre part, at least we agree on something. ;-)
CONFIG_NO_HZ_FULL booted with at least one nohz_full CPU. Said CPU gets into the kernel and stays there, not necessarily generating RCU callbacks. The other CPUs are very likely generating RCU callbacks. Because the nohz_full CPU is in the kernel, and because there are no scheduling-clock interrupts on that CPU, grace periods do not complete. Eventually, the callbacks from the other CPUs (and perhaps also some from the nohz_full CPU, for that matter) OOM the machine.
Now this scenario constitutes an abuse of CONFIG_NO_HZ_FULL, because it is intended for CPUs that execute either in userspace (in which case those CPUs are in extended quiescent states so that RCU can happily ignore them) or for real-time workloads with low CPU untilization (in which case RCU sees them go idle, which is also a quiescent state). But that won't stop people from abusing their kernels and complaining when things break.
This same thing can also happen without CONFIG_NO_HZ full, though the system has to work a bit harder. In this case, the CPU looping in the kernel has scheduling-clock interrupts, but if all it does is cond_resched(), RCU is never informed of any quiescent states. The whole point of this patch is to make those cond_resched() calls, which are quiescent states, visible to RCU.
Quoted 2 lines
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
I could get behind that. But by that definition, there is a lot of breakage in the current kernel, especially as we move to larger CPU counts.
Thanx, Paul
Quoted 15 lines · Mike Galbraith
Quoted 14 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
Quoted 6 lines
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Or put it all behind some debugging ifdef.
My first thought was to put it behind
CONFIG_NO_HZ_FULL, but everyone seems to be enabling that one.Not everybody, SUSE doesn't even have it enabled in factory.
OK, apologies for the over-generalization.
But you would think that I would have learned this lesson with CONFIG_RCU_FAST_NO_HZ, wouldn't you? :-/
Thanx, Paul
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
Quoted 6 lines
Quoted 1 line
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
Typical problem we faced in the past is in exit() path when multi thousands of files/sockets are rcu-freed, and qhimark is hit.
Huge latency alerts, as freeing 10000+ items takes a while (about 70 ns per item...)
Maybe close_files() should use a cond_resched_and_keep_rcu_queues_small_please() ;)
Quoted 15 lines · Eric Dumazet
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
Quoted 6 lines
Quoted 1 line
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
Quoted 8 lines
Typical problem we faced in the past is in
exit()path when multi thousands of files/sockets are rcu-freed, and qhimark is hit.Huge latency alerts, as freeing 10000+ items takes a while (about 70 ns per item...)
Maybe
close_files()should use acond_resched_and_keep_rcu_queues_small_please();)
That sort of approach would work for me. Over time, I would guess that the cond_resched_and_keep_rcu_queues_small_please() function would find its way to where it needed to be. ;-)
Thanx, Paul
Quoted 22 lines · Paul E. McKenney
Quoted 10 lines · Andi Kleen
Quoted 7 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
On the bizarre part, at least we agree on something. ;-)
CONFIG_NO_HZ_FULLbooted with at least onenohz_fullCPU. Said CPU gets into the kernel and stays there, not necessarily generating RCU callbacks. The other CPUs are very likely generating RCU callbacks. Because thenohz_fullCPU is in the kernel, and because there are no scheduling-clock interrupts on that CPU, grace periods do not complete. Eventually, the callbacks from the other CPUs (and perhaps also some from thenohz_fullCPU, for that matter) OOM the machine.
Quoted 7 lines
Now this scenario constitutes an abuse of
CONFIG_NO_HZ_FULL, because it is intended for CPUs that execute either in userspace (in which case those CPUs are in extended quiescent states so that RCU can happily ignore them) or for real-time workloads with low CPU untilization (in which case RCU sees them go idle, which is also a quiescent state). But that won't stop people from abusing their kernels and complaining when things break.
IMHO, those people can keep the pieces.
I don't even enable RCU_BOOST in -rt kernels, because that safety net has a price. The instant Joe User picks up the -rt shovel, it's his grave, and he gets to do the digging. Instead of trying to save his bacon, I hand him a slightly better shovel, let him prioritize all kthreads including workqueues. Joe can dig all he wants to, and it's on him, I just make sure he has the means to bury himself properly :)
Quoted 7 lines
This same thing can also happen without
CONFIG_NO_HZfull, though the system has to work a bit harder. In this case, the CPU looping in the kernel has scheduling-clock interrupts, but if all it does iscond_resched(), RCU is never informed of any quiescent states. The whole point of this patch is to make thosecond_resched()calls, which are quiescent states, visible to RCU.
Quoted 6 lines
Quoted 2 lines
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
I could get behind that. But by that definition, there is a lot of breakage in the current kernel, especially as we move to larger CPU counts.
Not only larger CPU counts: skipping the -rq clock update on wakeup (cycle saving optimization) turned out to be deadly to boxen with a zillion disks because our wakeup latency can be so incredibly horrible that falsely attributing wakeup latency to the next task to run (watchdog) resulted in it being throttled for long enough that big IO boxen panicked during boot.
The root cause of that wasn't the optimization, the root was the horrific amounts of time we can spend locked up in the kernel.
-Mike
Quoted 11 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.Or put it all behind some debugging ifdef.
Again I am fully on Andi's side here. Please remove these frequent calls to cond_resched. If one wants a fully preemptable kernel then please use CONFIG_PREEMPT.
Quoted 9 lines · Christoph Lameter
Quoted 7 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
cond_resched()is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads.
Quoted 8 lines
Quoted 4 lines
If you really wanted to do this I think you would first need to define a
cond_resched_i_am_not_fast()or somesuch.Or put it all behind some debugging ifdef.
Again I am fully on Andi's side here. Please remove these frequent calls to
cond_resched. If one wants a fully preemptable kernel then please useCONFIG_PREEMPT.
That is a separate issue, but unnecessary calls to cond_resched() should of course be removed -- no argument there.
Thanx, Paul
Quoted 31 lines · Mike Galbraith
Quoted 22 lines · Paul E. McKenney
Quoted 10 lines · Andi Kleen
Quoted 7 lines · Paul E. McKenney
Quoted 4 lines · Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally.
OOM prevention should count for something, I would hope.
OOM in what scenario? This is getting bizarre.
On the bizarre part, at least we agree on something. ;-)
CONFIG_NO_HZ_FULLbooted with at least onenohz_fullCPU. Said CPU gets into the kernel and stays there, not necessarily generating RCU callbacks. The other CPUs are very likely generating RCU callbacks. Because thenohz_fullCPU is in the kernel, and because there are no scheduling-clock interrupts on that CPU, grace periods do not complete. Eventually, the callbacks from the other CPUs (and perhaps also some from thenohz_fullCPU, for that matter) OOM the machine.Quoted 7 lines
Now this scenario constitutes an abuse of
CONFIG_NO_HZ_FULL, because it is intended for CPUs that execute either in userspace (in which case those CPUs are in extended quiescent states so that RCU can happily ignore them) or for real-time workloads with low CPU untilization (in which case RCU sees them go idle, which is also a quiescent state). But that won't stop people from abusing their kernels and complaining when things break.
Quoted 8 lines
IMHO, those people can keep the pieces.
I don't even enable
RCU_BOOSTin -rt kernels, because that safety net has a price. The instant Joe User picks up the -rt shovel, it's his grave, and he gets to do the digging. Instead of trying to save his bacon, I hand him a slightly better shovel, let him prioritize all kthreads including workqueues. Joe can dig all he wants to, and it's on him, I just make sure he has the means to bury himself properly :)
One of the nice things about NO_HZ_FULL is that it should reduce the need for RCU_BOOST. One of the purposes of RCU_BOOST is for the guy who has an infinite-loop bug in a high-priority RT-priority process, because such processes can starve out low-priority RCU readers. But with a properly configured NO_HZ_FULL system, the low-priority processes aren't sharing CPUs with the RT-priority processes.
In fact, it might be worth making RCU_BOOST depend on !NO_HZ_FULL in -rt.
Quoted 21 lines
Quoted 7 lines
This same thing can also happen without
CONFIG_NO_HZfull, though the system has to work a bit harder. In this case, the CPU looping in the kernel has scheduling-clock interrupts, but if all it does iscond_resched(), RCU is never informed of any quiescent states. The whole point of this patch is to make thosecond_resched()calls, which are quiescent states, visible to RCU.Quoted 6 lines
Quoted 2 lines
If something keeps looping forever in the kernel creating RCU callbacks without any real quiescent states it's simply broken.
I could get behind that. But by that definition, there is a lot of breakage in the current kernel, especially as we move to larger CPU counts.
Not only larger CPU counts: skipping the -rq clock update on wakeup (cycle saving optimization) turned out to be deadly to boxen with a zillion disks because our wakeup latency can be so incredibly horrible that falsely attributing wakeup latency to the next task to run (watchdog) resulted in it being throttled for long enough that big IO boxen panicked during boot.
Quoted 2 lines
The root cause of that wasn't the optimization, the root was the horrific amounts of time we can spend locked up in the kernel.
Completely agreed!
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
That is a separate issue, but unnecessary calls to
cond_resched()should of course be removed -- no argument there.
It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the latencies that we are trying to reduce.
Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
That is a separate issue, but unnecessary calls to
cond_resched()should of course be removed -- no argument there.It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the latencies that we are trying to reduce.
That is on battle in a larger war. Another battle is to keep big systems from choking and dying. We need to win both battles, though I understand that you are focused primarily on latency reduction. I don't get that luxury because RCU is used across the board.
Thanx, Paul
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
That is a separate issue, but unnecessary calls to
cond_resched()should of course be removed -- no argument there.
Quoted 6 lines
Quoted 3 lines
It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the latencies that we are trying to reduce.
That is on battle in a larger war. Another battle is to keep big systems from choking and dying. We need to win both battles, though I understand
If we really need cond_resched() now to stop systems from dying we did a really poor job at designing Linux. I hope it's not true, that would be really bad and point to some severe design problems elsewhere.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
That is a separate issue, but unnecessary calls to
cond_resched()should of course be removed -- no argument there.It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the latencies that we are trying to reduce.
Quoted 7 lines
Quoted 2 lines
That is on battle in a larger war. Another battle is to keep big systems from choking and dying. We need to win both battles, though I understand
If we really need
cond_resched()now to stop systems from dying we did a really poor job at designing Linux. I hope it's not true, that would be really bad and point to some severe design problems elsewhere.
Or that systems are growing larger than anticipated by some of the designs. And there are some truly huge systems out there running Linux, as has been mentioned on this thread several times.
Thanx, Paul
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
That is a separate issue, but unnecessary calls to
cond_resched()should of course be removed -- no argument there.
Quoted 8 lines
Quoted 3 lines
It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the latencies that we are trying to reduce.
That is on battle in a larger war. Another battle is to keep big systems from choking and dying. We need to win both battles, though I understand that you are focused primarily on latency reduction. I don't get that luxury because RCU is used across the board.
At least reduce the increased latencies to a processor or put it somewhere outside of the regular execution paths. Sprinkling this throughout the kernel via a inline function that did not generate code in the non debug situation before is not that nice.
This looks very much like the CONFIG_PREEMPT problem in not so extreme form. Maybe we need to add another config option:
CONFIG_REALLY_REALLY_NO_PREEMPT
to get the fastest code possible and those cond_rescheds removed from the critical paths?
Or better
CONFIG_PREEMPT_HALF_WAY
to enable those cond_rescheds.
[email protected]
maintainerQuoted 5 lines · Christoph Lameter
This looks very much like the
CONFIG_PREEMPTproblem in not so extreme form. Maybe we need to add another config option:
CONFIG_REALLY_REALLY_NO_PREEMPT
Quoted 8 lines
to get the fastest code possible and those
cond_reschedsremoved from the critical paths?Or better
CONFIG_PREEMPT_HALF_WAYto enable those
cond_rescheds.
That much actually does seem quite reasonable: making cond_resched() do non-trivial work ought to have a config option to disable it.
Quoted 9 lines · On Thu, Jun 19, 2014 at 02:32:03PM -0700, [email protected] wrote:
Quoted 8 lines · Christoph Lameter
This looks very much like the
CONFIG_PREEMPTproblem in not so extreme form. Maybe we need to add another config option:
CONFIG_REALLY_REALLY_NO_PREEMPTto get the fastest code possible and those
cond_reschedsremoved from the critical paths?
Quoted 8 lines
Quoted 5 lines
Or better
CONFIG_PREEMPT_HALF_WAYto enable those
cond_rescheds.That much actually does seem quite reasonable: making
cond_resched()do non-trivial work ought to have a config option to disable it.
I am putting together patches based on Eric Dumazet's suggestion and on a variant of the above approach. However, I do expect the distros to starve to death between those two bales of hay. But perhaps the actual patches will inspire a bit of light to go with this thread's heat and smoke.
Thanx, Paul
Quoted 4 lines · Paul E. McKenney
I am putting together patches based on Eric Dumazet's suggestion and on a variant of the above approach. However, I do expect the distros to starve to death between those two bales of hay. But perhaps the actual patches will inspire a bit of light to go with this thread's heat and smoke.
Yep. Note that the large systems are only a few and the smaller systems are many. You are penalizing the overwhelming majority for support of a few systems.
Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Paul E. McKenney
I am putting together patches based on Eric Dumazet's suggestion and on a variant of the above approach. However, I do expect the distros to starve to death between those two bales of hay. But perhaps the actual patches will inspire a bit of light to go with this thread's heat and smoke.
Yep. Note that the large systems are only a few and the smaller systems are many. You are penalizing the overwhelming majority for support of a few systems.
And the fraction of the small systems that care that much about latency is also quite small.
Of course, if you have performance results that show otherwise, please share them.
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
And the fraction of the small systems that care that much about latency is also quite small.
This also impacts general performance of course not only latency.
Quoted 2 lines
Of course, if you have performance results that show otherwise, please share them.
We are discussing a performance problem right here.
Quoted 11 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
And the fraction of the small systems that care that much about latency is also quite small.
This also impacts general performance of course not only latency.
Quoted 2 lines
Of course, if you have performance results that show otherwise, please share them.
We are discussing a performance problem right here.
We are engaging in a lot of -speculation- about an -alleged- performance problem. I have posted a fix to the actual performance problem, but the guy reporting the original report of that problem has not yet reported the performance of the fix. Which is fine, as I am getting ready to post an updated fix, and besides which he might well have higher-priority tasks requiring his attention. But without that performance information, we are all just engaging in baseless speculation.
Thanx, Paul
Quoted 13 lines · Paul E. McKenney
Quoted 11 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
And the fraction of the small systems that care that much about latency is also quite small.
This also impacts general performance of course not only latency.
Quoted 2 lines
Of course, if you have performance results that show otherwise, please share them.
We are discussing a performance problem right here.
Quoted 7 lines
We are engaging in a lot of -speculation- about an -alleged- performance problem. I have posted a fix to the actual performance problem, but the guy reporting the original report of that problem has not yet reported the performance of the fix. Which is fine, as I am getting ready to post an updated fix, and besides which he might well have higher-priority tasks requiring his attention. But without that performance information, we are all just engaging in baseless speculation.
The patch helps, but does not get us back to where we were before:
Baseline: 3c8fb5044 with ac1bea85781e9004d reverted:
root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 ... average:57710064
3c8fb5044 with Paul's early version with the magic 7:
root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 ... average:48363361
It was down at 43M or 44M without the new patch.
full spew:
3c8fb50445833b93f69b6b703a29aae3523cad0c
with ac1bea85781e9004d reverted:
root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 warmup min:366329 max:421686 total:63727790 min:358758 max:406036 total:62561168 min:298435 max:340355 total:51711000 min:343707 max:383812 total:59554601 min:310652 max:356571 total:54016014 min:337738 max:382557 total:59146223 measurement min:317628 max:363489 total:55230696 min:341875 max:392621 total:59910047 min:331979 max:376582 total:58074347 min:342835 max:388481 total:59644247 min:327282 max:374283 total:56897588 min:321668 max:373622 total:55799061 min:336575 max:387027 total:58948777 min:350269 max:397098 total:61510405 min:336949 max:382367 total:58640224 min:327913 max:371026 total:56833639 min:322377 max:375011 total:56923970 min:354508 max:399130 total:61628878 min:319998 max:367493 total:55624467 min:337322 max:379689 total:58425775 min:325701 max:369950 total:56382432 min:318423 max:364997 total:55662597 min:332930 max:374174 total:57881582 min:324787 max:371432 total:56881287 min:335136 max:381850 total:58661101 min:329253 max:377998 total:58146731 min:353550 max:399807 total:61031126 min:312443 max:358633 total:55092296 min:344435 max:396344 total:61129639 min:332426 max:394539 total:58307766 min:322235 max:367385 total:56477056 min:326722 max:362761 total:55421905 min:318183 max:366510 total:56260638 min:321812 max:372084 total:56639167 min:328016 max:379484 total:57794343 min:314929 max:364644 total:55440152 average:57710064
3c8fb50445833b93f69b6b703a29aae3523cad0c
with paul's early version and the magic 7:
root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 testcase:Separate file open/close warmup min:310891 max:364268 total:54015547 min:283886 max:333177 total:49992270 min:275451 max:314699 total:47775422 min:275164 max:316351 total:47720614 min:283434 max:326614 total:49156206 min:276475 max:320313 total:48469874 measurement min:274589 max:323239 total:48149462 min:280194 max:328827 total:49338289 min:277830 max:327060 total:48962337 min:281689 max:325917 total:49211850 min:271332 max:308350 total:46813631 min:275384 max:319016 total:47366390 min:276728 max:318277 total:47752679 min:275616 max:317111 total:47955871 min:278882 max:319082 total:47864449 min:267076 max:312427 total:47225886 min:279400 max:322798 total:48561113 min:279140 max:317367 total:47762975 min:278772 max:325575 total:49001990 min:271660 max:311780 total:47349871 min:283508 max:326299 total:49516078 min:284946 max:327525 total:49468752 min:271924 max:316507 total:47867902 min:279669 max:313937 total:47582153 min:272887 max:319545 total:47725521 min:281685 max:323791 total:48620478 min:280714 max:324849 total:48804594 min:283307 max:327424 total:49153787 min:280186 max:323924 total:48618088 min:270821 max:311985 total:47194656 min:282145 max:327910 total:48728398 min:276035 max:326523 total:48895232 min:275433 max:317441 total:48054952 min:276840 max:320486 total:48297727 min:280297 max:320323 total:48736783 min:286099 max:327522 total:50318962 average:48363361
Quoted 29 lines · Dave Hansen
Quoted 13 lines · Paul E. McKenney
Quoted 11 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
And the fraction of the small systems that care that much about latency is also quite small.
This also impacts general performance of course not only latency.
Quoted 2 lines
Of course, if you have performance results that show otherwise, please share them.
We are discussing a performance problem right here.
Quoted 7 lines
We are engaging in a lot of -speculation- about an -alleged- performance problem. I have posted a fix to the actual performance problem, but the guy reporting the original report of that problem has not yet reported the performance of the fix. Which is fine, as I am getting ready to post an updated fix, and besides which he might well have higher-priority tasks requiring his attention. But without that performance information, we are all just engaging in baseless speculation.
The patch helps, but does not get us back to where we were before:
Baseline:
3c8fb5044withac1bea85781e9004dreverted:root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 ... average:57710064
Quoted 6 lines
3c8fb5044with Paul's early version with the magic 7:root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 ... average:48363361It was down at 43M or 44M without the new patch.
Thank you for the info, Dave!
Thanx, Paul
Quoted 90 lines
full spew:
3c8fb50445833b93f69b6b703a29aae3523cad0c
withac1bea85781e9004dreverted:root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 warmup min:366329 max:421686 total:63727790 min:358758 max:406036 total:62561168 min:298435 max:340355 total:51711000 min:343707 max:383812 total:59554601 min:310652 max:356571 total:54016014 min:337738 max:382557 total:59146223 measurement min:317628 max:363489 total:55230696 min:341875 max:392621 total:59910047 min:331979 max:376582 total:58074347 min:342835 max:388481 total:59644247 min:327282 max:374283 total:56897588 min:321668 max:373622 total:55799061 min:336575 max:387027 total:58948777 min:350269 max:397098 total:61510405 min:336949 max:382367 total:58640224 min:327913 max:371026 total:56833639 min:322377 max:375011 total:56923970 min:354508 max:399130 total:61628878 min:319998 max:367493 total:55624467 min:337322 max:379689 total:58425775 min:325701 max:369950 total:56382432 min:318423 max:364997 total:55662597 min:332930 max:374174 total:57881582 min:324787 max:371432 total:56881287 min:335136 max:381850 total:58661101 min:329253 max:377998 total:58146731 min:353550 max:399807 total:61031126 min:312443 max:358633 total:55092296 min:344435 max:396344 total:61129639 min:332426 max:394539 total:58307766 min:322235 max:367385 total:56477056 min:326722 max:362761 total:55421905 min:318183 max:366510 total:56260638 min:321812 max:372084 total:56639167 min:328016 max:379484 total:57794343 min:314929 max:364644 total:55440152 average:57710064
3c8fb50445833b93f69b6b703a29aae3523cad0c
with paul's early version and the magic 7:root@bigbox:~/will-it-scale# ./open1_processes -t 160 -s 30 testcase:Separate file open/close warmup min:310891 max:364268 total:54015547 min:283886 max:333177 total:49992270 min:275451 max:314699 total:47775422 min:275164 max:316351 total:47720614 min:283434 max:326614 total:49156206 min:276475 max:320313 total:48469874 measurement min:274589 max:323239 total:48149462 min:280194 max:328827 total:49338289 min:277830 max:327060 total:48962337 min:281689 max:325917 total:49211850 min:271332 max:308350 total:46813631 min:275384 max:319016 total:47366390 min:276728 max:318277 total:47752679 min:275616 max:317111 total:47955871 min:278882 max:319082 total:47864449 min:267076 max:312427 total:47225886 min:279400 max:322798 total:48561113 min:279140 max:317367 total:47762975 min:278772 max:325575 total:49001990 min:271660 max:311780 total:47349871 min:283508 max:326299 total:49516078 min:284946 max:327525 total:49468752 min:271924 max:316507 total:47867902 min:279669 max:313937 total:47582153 min:272887 max:319545 total:47725521 min:281685 max:323791 total:48620478 min:280714 max:324849 total:48804594 min:283307 max:327424 total:49153787 min:280186 max:323924 total:48618088 min:270821 max:311985 total:47194656 min:282145 max:327910 total:48728398 min:276035 max:326523 total:48895232 min:275433 max:317441 total:48054952 min:276840 max:320486 total:48297727 min:280297 max:320323 total:48736783 min:286099 max:327522 total:50318962 average:48363361