rcu/rcuperf: Add kfree_rcu() performance Tests
Subsystems (4)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/5 rcu/rcuperf: Add kfree_rcu() performance Tests
This test runs kfree_rcu() in a loop to measure performance of the new kfree_rcu() batching functionality.
The following table shows results when booting with arguments:
rcuperf.kfree_loops=20000 rcuperf.kfree_alloc_num=8000 rcuperf.kfree_rcu_test=1
In addition, rcuperf.kfree_no_batch is used to toggle the batching of kfree_rcu()s for a test run.
patch applied GPs time (seconds)
yes 1732 14.5
no 9133 11.5
On a 16 CPU system with the above boot parameters, we see that the total number of grace periods that elapse during the test drops from 9133 when not batching to 1732 when batching (a 5X improvement). The kfree_rcu() flood itself slows down a bit when batching, though, as shown.
Note that the active memory consumption during the kfree_rcu() flood does increase to around 200-250MB due to the batching (from around 50MB without batching). However, this memory consumption is relatively constant. In other words, the system is able to keep up with the kfree_rcu() load. The memory consumption comes down considerably if KFREE_DRAIN_JIFFIES is increased from HZ/50 to HZ/80.
Also, when running the test, please disable CONFIG_DEBUG_PREEMPT and CONFIG_PROVE_RCU for realistic comparisons with/without batching.
--- .../admin-guide/kernel-parameters.txt | 17 ++ kernel/rcu/rcuperf.c | 181 +++++++++++++++++- 2 files changed, 190 insertions(+), 8 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +17 -0
@@ -3896,6 +3896,23 @@ test until boot completes in order to avoid interference. + rcuperf.kfree_rcu_test= [KNL] + Set to measure performance of kfree_rcu() flooding. + + rcuperf.kfree_nthreads= [KNL] + The number of threads running loops of kfree_rcu(). + + rcuperf.kfree_alloc_num= [KNL] + Number of allocations and frees done in an iteration. + + rcuperf.kfree_loops= [KNL] + Number of loops doing rcuperf.kfree_alloc_num number + of allocations and frees. + + rcuperf.kfree_no_batch= [KNL] + Use the non-batching (less efficient) version of kfree_rcu(). + This is useful for comparing with the batched version. + rcuperf.nreaders= [KNL] Set number of RCU readers. The value -1 selects N, where N is the number of CPUs. A value
kernel/rcu/rcuperf.c +173 -8
227 lines not on this page · raw message
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerQuoted 6 lines · Joel Fernandes (Google)
This test runs
kfree_rcu()in a loop to measure performance of the newkfree_rcu()batching functionality.The following table shows results when booting with arguments:
rcuperf.kfree_loops=20000 rcuperf.kfree_alloc_num=8000 rcuperf.kfree_rcu_test=1
Quoted 6 lines
In addition, rcuperf.kfree_no_batch is used to toggle the batching of kfree_rcu()s for a test run.
patch applied GPs time (seconds)
yes 1732 14.5
no 9133 11.5
This is really "rcuperf.kfree_no_batch" rather than "patch applied", right? (Yes, we did discuss this last time around, but this table combined with the prior paragraph is still ambiguous.) Please make it unambiguous. One way to do that is as follows:
------------------------------------------------------------------------
The following table shows results when booting with arguments:
rcuperf.kfree_loops=20000 rcuperf.kfree_alloc_num=8000 rcuperf.kfree_rcu_test=1 rcuperf.kfree_no_batch=X
rcuperf.kfree_no_batch=X # Grace Periods Test Duration (s)
X=1 (old behavior) 9133 11.5 X=0 (new behavior) 1732 14.5
------------------------------------------------------------------------
Quoted 4 lines
On a 16 CPU system with the above boot parameters, we see that the total number of grace periods that elapse during the test drops from 9133 when not batching to 1732 when batching (a 5X improvement). The
kfree_rcu()flood itself slows down a bit when batching, though, as shown.
This last sentence would be more clear as something like: "However, use of batching increases the duration of the kfree_rcu()-flood test."
Quoted 6 lines
Note that the active memory consumption during the
kfree_rcu()flood does increase to around 200-250MB due to the batching (from around 50MB without batching). However, this memory consumption is relatively constant. In other words, the system is able to keep up with thekfree_rcu()load. The memory consumption comes down considerably ifKFREE_DRAIN_JIFFIESis increased from HZ/50 to HZ/80.
That would be a decrease rather than an increase in KFREE_DRAIN_JIFFIES, correct?
This would also be a good place to mention that a later patch will decrease consumption, but that is strictly optional. However, you did introduce the topic of changing KFREE_DRAIN_JIFFIES, so if a later patch changes this value, this would be an excellent place to mention this.
Quoted 4 lines
Also, when running the test, please disable
CONFIG_DEBUG_PREEMPTandCONFIG_PROVE_RCUfor realistic comparisons with/without batching.Signed-off-by Joel Fernandes (Google) <[email protected]>
And the code is much better! Just one duplication-avoidance nit below. Plus a thought for a future patch.
Thanx, Paul
Quoted 124 lines
--- .../admin-guide/kernel-parameters.txt | 17 ++ kernel/rcu/rcuperf.c | 181 +++++++++++++++++- 2 files changed, 190 insertions(+), 8 deletions(-)Documentation/admin-guide/kernel-parameters.txt +17
-0@@ -3896,6 +3896,23 @@ test until boot completes in order to avoid interference. + rcuperf.kfree_rcu_test= [KNL] + Set to measure performance of kfree_rcu() flooding. + + rcuperf.kfree_nthreads= [KNL] + The number of threads running loops of kfree_rcu(). + + rcuperf.kfree_alloc_num= [KNL] + Number of allocations and frees done in an iteration. + + rcuperf.kfree_loops= [KNL] + Number of loops doing rcuperf.kfree_alloc_num number + of allocations and frees. + + rcuperf.kfree_no_batch= [KNL] + Use the non-batching (less efficient) version of kfree_rcu(). + This is useful for comparing with the batched version. + rcuperf.nreaders= [KNL] Set number of RCU readers. The value -1 selects N, where N is the number of CPUs. A valuekernel/rcu/rcuperf.c +36
-8@@ -86,6 +86,7 @@ torture_param(bool, shutdown, RCUPERF_SHUTDOWN, "Shutdown at end of performance tests.");torture_param (int, verbose,1 ,"Enable verbose debugging printk()s" );torture_param (int, writer_holdoff,0 ,"Holdoff (us) between GPs, zero to disable" ); +torture_param (int, kfree_rcu_test,0 ,"Do we run a kfree_rcu() perf test?" );static char *perf_type ="rcu" ;module_param (perf_type, charp,0444 );@@ -105,8 +106,8 @@ static atomic_t n_rcu_perf_writer_finished;static wait_queue_head_t shutdown_wq;static u64 t_rcu_perf_writer_started;static u64 t_rcu_perf_writer_finished; -static unsigned long b_rcu_perf_writer_started; -static unsigned long b_rcu_perf_writer_finished; +static unsigned long b_rcu_gp_test_started; +static unsigned long b_rcu_gp_test_finished; static DEFINE_PER_CPU(atomic_t, n_async_inflight);#define MAX_MEAS 10000@@ -378,10 +379,10 @@ rcu_perf_writer(void *arg)if (atomic_inc_return (&n_rcu_perf_writer_started) >= nrealwriters) { t_rcu_perf_writer_started = t;if (gp_exp) { - b_rcu_perf_writer_started = + b_rcu_gp_test_started = cur_ops->exp_completed () /2 ; }else { - b_rcu_perf_writer_started = cur_ops->get_gp_seq (); + b_rcu_gp_test_started = cur_ops->get_gp_seq (); } }@@ -429,10 +430,10 @@ rcu_perf_writer(void *arg)PERFOUT_STRING ("Test complete" ); t_rcu_perf_writer_finished = t;if (gp_exp) { - b_rcu_perf_writer_finished = + b_rcu_gp_test_finished = cur_ops->exp_completed () /2 ; }else { - b_rcu_perf_writer_finished = + b_rcu_gp_test_finished = cur_ops->get_gp_seq (); }if (shutdown) {@@ -515,8 +516,8 @@ rcu_perf_cleanup(void) t_rcu_perf_writer_finished - t_rcu_perf_writer_started, ngps, -rcuperf_seq_diff (b_rcu_perf_writer_finished, - b_rcu_perf_writer_started)); +rcuperf_seq_diff (b_rcu_gp_test_finished, + b_rcu_gp_test_started));for (i =0 ; i < nrealwriters; i++) {if (!writer_durations)break ;@@ -584,6 +585,167 @@ rcu_perf_shutdown(void *arg)return -EINVAL ; } +/* +* kfree_rcu() performance tests: Start a kfree_rcu() loop on all CPUs for number +* of iterations and measure total time and number of GP for all iterations to complete. +*/ + +torture_param (int, kfree_nthreads,-1 ,"Number of threads running loops of kfree_rcu()." ); +torture_param (int, kfree_alloc_num,8000 ,"Number of allocations and frees done in an iteration." ); +torture_param (int, kfree_loops,10 ,"Number of loops doing kfree_alloc_num allocations and frees." ); +torture_param (int, kfree_no_batch,0 ,"Use the non-batching (slower) version of kfree_rcu()." ); + +static struct task_struct **kfree_reader_tasks; +static int kfree_nrealthreads; +static atomic_t n_kfree_perf_thread_started; +static atomic_t n_kfree_perf_thread_ended; + +struct kfree_obj { +char kfree_obj[8 ]; +struct rcu_head rh; +}; + +static int +kfree_perf_thread (void *arg) +{ +int i, loop =0 ; +long me = (long )arg; +struct kfree_obj *alloc_ptr; +u64 start_time, end_time;
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 624
Quoted 10 lines
kernel/rcu/rcuperf.c +10
-0+ +VERBOSE_PERFOUT_STRING ("kfree_perf_thread task started" ); +set_cpus_allowed_ptr (current,cpumask_of (me % nr_cpu_ids)); +set_user_nice (current,MAX_NICE ); + + start_time =ktime_get_mono_fast_ns (); + +if (atomic_inc_return (&n_kfree_perf_thread_started) >= kfree_nrealthreads) { +if (gp_exp) + b_rcu_gp_test_started = cur_ops->exp_completed () /2 ;
At some point, it would be good to use the new grace-period sequence-counter functions (rcuperf_seq_diff(), for example) instead of the open-coded division by 2. I freely admit that you are just copying my obsolete hack in this case, so not needed in this patch.
Quoted 9 lines
kernel/rcu/rcuperf.c +9
-0+else + b_rcu_gp_test_started = cur_ops->get_gp_seq (); + } + + do { + for (i = 0; i < kfree_alloc_num; i++) { + alloc_ptr = kmalloc(sizeof(struct kfree_obj), GFP_KERNEL); + if (!alloc_ptr) + return -ENOMEM;
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 643
Quoted 10 lines
kernel/rcu/rcuperf.c +10
-0+ +if (!kfree_no_batch) { +kfree_rcu (alloc_ptr, rh); + }else { +rcu_callback_t cb; + + cb = (rcu_callback_t )(unsigned long )offsetof(struct kfree_obj , rh); +kfree_call_rcu_nobatch (&(alloc_ptr->rh), cb); + } + }
Nice, much simpler than the earlier version!
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 652
Quoted 8 lines
kernel/rcu/rcuperf.c +8
-0+cond_resched (); + }while (!torture_must_stop () && ++loop < kfree_loops); + +if (atomic_inc_return (&n_kfree_perf_thread_ended) >= kfree_nrealthreads) { + end_time =ktime_get_mono_fast_ns (); + +if (gp_exp) + b_rcu_gp_test_finished = cur_ops->exp_completed () /2 ;
Same here on open-coded division by 2.
Quoted 41 lines
kernel/rcu/rcuperf.c +41
-0+else + b_rcu_gp_test_finished = cur_ops->get_gp_seq (); + +pr_alert ("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n" , + (unsigned long long )(end_time - start_time), kfree_loops, +rcuperf_seq_diff (b_rcu_gp_test_finished, b_rcu_gp_test_started)); +if (shutdown) { +smp_mb ();/* Assign before wake. */ +wake_up (&shutdown_wq); + } + } + +torture_kthread_stopping ("kfree_perf_thread" ); +return 0 ; +} + +static void +kfree_perf_cleanup (void ) +{ +int i; + +if (torture_cleanup_begin ()) +return ; + +if (kfree_reader_tasks) { +for (i =0 ; i < kfree_nrealthreads; i++) +torture_stop_kthread (kfree_perf_thread, + kfree_reader_tasks[i]); +kfree (kfree_reader_tasks); + } + +torture_cleanup_end (); +} + +/* +* shutdown kthread. Just waits to be awakened, then shuts down system. +*/ +static int +kfree_perf_shutdown (void *arg) +{ + do {
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 703
Quoted 10 lines
kernel/rcu/rcuperf.c +10
-0+wait_event (shutdown_wq, +atomic_read (&n_kfree_perf_thread_ended) >= + kfree_nrealthreads); + }while (atomic_read (&n_kfree_perf_thread_ended) < kfree_nrealthreads); + +smp_mb ();/* Wake before output. */ + +kfree_perf_cleanup (); +kernel_power_off (); +return -EINVAL ;
These last four lines should be combined with those of rcu_perf_shutdown(). Actually, you could fold the two functions together with only a pair of arguments and two one-line wrapper functions, which would be even better.
Quoted 61 lines
kernel/rcu/rcuperf.c +48
-0+} + +static int __init +kfree_perf_init (void ) +{ +long i; +int firsterr =0 ; + + kfree_nrealthreads =compute_real (kfree_nthreads); +/* Start up the kthreads. */ +if (shutdown) { +init_waitqueue_head (&shutdown_wq); + firsterr =torture_create_kthread (kfree_perf_shutdown,NULL , + shutdown_task); +if (firsterr) + goto unwind; +schedule_timeout_uninterruptible (1 ); + } + + kfree_reader_tasks =kcalloc (kfree_nrealthreads,sizeof (kfree_reader_tasks[0 ]), +GFP_KERNEL ); +if (kfree_reader_tasks ==NULL ) { + firsterr = -ENOMEM ; + goto unwind; + } + +for (i =0 ; i < kfree_nrealthreads; i++) { + firsterr =torture_create_kthread (kfree_perf_thread, (void *)i, + kfree_reader_tasks[i]); +if (firsterr) + goto unwind; + } + +while (atomic_read (&n_kfree_perf_thread_started) < kfree_nrealthreads) +schedule_timeout_uninterruptible (1 ); + +torture_init_end (); +return 0 ; + +unwind: +torture_init_end (); +kfree_perf_cleanup (); +return firsterr; +} +static int __init rcu_perf_init (void ) {@@ -616,6 +778,9 @@ rcu_perf_init(void)if (cur_ops->init) cur_ops->init (); +if (kfree_rcu_test) +return kfree_perf_init (); + nrealwriters =compute_real (nwriters); nrealreaders =compute_real (nreaders);atomic_set (&n_rcu_perf_reader_started,0 );Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Quoted 19 lines · Paul E. McKenney
Quoted 6 lines · Joel Fernandes (Google)
This test runs
kfree_rcu()in a loop to measure performance of the newkfree_rcu()batching functionality.The following table shows results when booting with arguments:
rcuperf.kfree_loops=20000 rcuperf.kfree_alloc_num=8000 rcuperf.kfree_rcu_test=1Quoted 6 lines
In addition, rcuperf.kfree_no_batch is used to toggle the batching of kfree_rcu()s for a test run.
patch applied GPs time (seconds)
yes 1732 14.5
no 9133 11.5This is really "rcuperf.kfree_no_batch" rather than "patch applied", right? (Yes, we did discuss this last time around, but this table combined with the prior paragraph is still ambiguous.) Please make it unambiguous. One way to do that is as follows:
Quoted 8 lines
------------------------------------------------------------------------
The following table shows results when booting with arguments:
rcuperf.kfree_loops=20000 rcuperf.kfree_alloc_num=8000 rcuperf.kfree_rcu_test=1 rcuperf.kfree_no_batch=Xrcuperf.kfree_no_batch=X # Grace Periods Test Duration (s)
X=1 (old behavior) 9133 11.5 X=0 (new behavior) 1732 14.5
Yes you are right, will fix. The reason I changed it to 'patch applied' is because the last patch in the series removes kfree_no_batch. Will fix! thanks!
Quoted 15 lines
Quoted 4 lines
On a 16 CPU system with the above boot parameters, we see that the total number of grace periods that elapse during the test drops from 9133 when not batching to 1732 when batching (a 5X improvement). The
kfree_rcu()flood itself slows down a bit when batching, though, as shown.This last sentence would be more clear as something like: "However, use of batching increases the duration of the kfree_rcu()-flood test."
Quoted 6 lines
Note that the active memory consumption during the
kfree_rcu()flood does increase to around 200-250MB due to the batching (from around 50MB without batching). However, this memory consumption is relatively constant. In other words, the system is able to keep up with thekfree_rcu()load. The memory consumption comes down considerably ifKFREE_DRAIN_JIFFIESis increased from HZ/50 to HZ/80.
Quoted 7 lines
That would be a decrease rather than an increase in
KFREE_DRAIN_JIFFIES, correct?This would also be a good place to mention that a later patch will decrease consumption, but that is strictly optional. However, you did introduce the topic of changing
KFREE_DRAIN_JIFFIES, so if a later patch changes this value, this would be an excellent place to mention this.
Fixed.
[snip]
Quoted 34 lines
Quoted 34 lines
kernel/rcu/rcuperf.c +34
-0+/* +* kfree_rcu() performance tests: Start a kfree_rcu() loop on all CPUs for number +* of iterations and measure total time and number of GP for all iterations to complete. +*/ + +torture_param (int, kfree_nthreads,-1 ,"Number of threads running loops of kfree_rcu()." ); +torture_param (int, kfree_alloc_num,8000 ,"Number of allocations and frees done in an iteration." ); +torture_param (int, kfree_loops,10 ,"Number of loops doing kfree_alloc_num allocations and frees." ); +torture_param (int, kfree_no_batch,0 ,"Use the non-batching (slower) version of kfree_rcu()." ); + +static struct task_struct **kfree_reader_tasks; +static int kfree_nrealthreads; +static atomic_t n_kfree_perf_thread_started; +static atomic_t n_kfree_perf_thread_ended; + +struct kfree_obj { +char kfree_obj[8 ]; +struct rcu_head rh; +}; + +static int +kfree_perf_thread (void *arg) +{ +int i, loop =0 ; +long me = (long )arg; +struct kfree_obj *alloc_ptr; +u64 start_time, end_time; + +VERBOSE_PERFOUT_STRING ("kfree_perf_thread task started" ); +set_cpus_allowed_ptr (current,cpumask_of (me % nr_cpu_ids)); +set_user_nice (current,MAX_NICE ); + + start_time =ktime_get_mono_fast_ns (); +
Quoted 8 lines
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 624
Quoted 3 lines
kernel/rcu/rcuperf.c +3
-0+if (atomic_inc_return (&n_kfree_perf_thread_started) >= kfree_nrealthreads) { +if (gp_exp) + b_rcu_gp_test_started = cur_ops->exp_completed () /2 ;At some point, it would be good to use the new grace-period sequence-counter functions (
rcuperf_seq_diff(), for example) instead of the open-coded division by 2. I freely admit that you are just copying my obsolete hack in this case, so not needed in this patch.
But I am using rcu_seq_diff() below in the pr_alert().
Anyway, I agree this can be a follow-on since this pattern is borrowed from another part of rcuperf. However, I am also confused about the pattern itself.
If I understand, you are doing the "/ 2" because expedited_sequence progresses by 2 for every expedited batch.
But does rcu_seq_diff() really work on these expedited GP numbers, and will it be immune to changes in RCU_SEQ_STATE_MASK? Sorry for the silly questions, but admittedly I have not looked too much yet into expedited RCU so I could be missing the point.
Quoted 48 lines
Quoted 48 lines
kernel/rcu/rcuperf.c +48
-0+else + b_rcu_gp_test_finished = cur_ops->get_gp_seq (); + +pr_alert ("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n" , + (unsigned long long )(end_time - start_time), kfree_loops, +rcuperf_seq_diff (b_rcu_gp_test_finished, b_rcu_gp_test_started)); +if (shutdown) { +smp_mb ();/* Assign before wake. */ +wake_up (&shutdown_wq); + } + } + +torture_kthread_stopping ("kfree_perf_thread" ); +return 0 ; +} + +static void +kfree_perf_cleanup (void ) +{ +int i; + +if (torture_cleanup_begin ()) +return ; + +if (kfree_reader_tasks) { +for (i =0 ; i < kfree_nrealthreads; i++) +torture_stop_kthread (kfree_perf_thread, + kfree_reader_tasks[i]); +kfree (kfree_reader_tasks); + } + +torture_cleanup_end (); +} + +/* +* shutdown kthread. Just waits to be awakened, then shuts down system. +*/ +static int +kfree_perf_shutdown (void *arg) +{ +do { +wait_event (shutdown_wq, +atomic_read (&n_kfree_perf_thread_ended) >= + kfree_nrealthreads); + }while (atomic_read (&n_kfree_perf_thread_ended) < kfree_nrealthreads); + +smp_mb ();/* Wake before output. */ +
Quoted 8 lines
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 703
Quoted 3 lines
kernel/rcu/rcuperf.c +3
-0+kfree_perf_cleanup (); +kernel_power_off (); +return -EINVAL ;These last four lines should be combined with those of
rcu_perf_shutdown(). Actually, you could fold the two functions together with only a pair of arguments and two one-line wrapper functions, which would be even better.
But the cleanup() function is different in the 2 cases and will have to be passed in as a function pointer. I believe we discussed this last review as well.
thanks,
- Joel
Paul E. McKenney
maintainerQuoted 1 line · Joel Fernandes
On Wed, Aug 28, 2019 at 02:12:26PM -0700, Paul E. McKenney wrote:
[ . . . ]
Quoted 29 lines
Quoted 14 lines
Quoted 14 lines
kernel/rcu/rcuperf.c +14
-0+static int +kfree_perf_thread (void *arg) +{ +int i, loop =0 ; +long me = (long )arg; +struct kfree_obj *alloc_ptr; +u64 start_time, end_time; + +VERBOSE_PERFOUT_STRING ("kfree_perf_thread task started" ); +set_cpus_allowed_ptr (current,cpumask_of (me % nr_cpu_ids)); +set_user_nice (current,MAX_NICE ); + + start_time =ktime_get_mono_fast_ns (); +Quoted 8 lines
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 624
Quoted 3 lines
kernel/rcu/rcuperf.c +3
-0+if (atomic_inc_return (&n_kfree_perf_thread_started) >= kfree_nrealthreads) { +if (gp_exp) + b_rcu_gp_test_started = cur_ops->exp_completed () /2 ;At some point, it would be good to use the new grace-period sequence-counter functions (
rcuperf_seq_diff(), for example) instead of the open-coded division by 2. I freely admit that you are just copying my obsolete hack in this case, so not needed in this patch.But I am using
rcu_seq_diff()below in thepr_alert().Anyway, I agree this can be a follow-on since this pattern is borrowed from another part of rcuperf. However, I am also confused about the pattern itself.
Quoted 7 lines
If I understand, you are doing the "/ 2" because
expedited_sequenceprogresses by 2 for every expedited batch.But does
rcu_seq_diff()really work on these expedited GP numbers, and will it be immune to changes inRCU_SEQ_STATE_MASK? Sorry for the silly questions, but admittedly I have not looked too much yet into expedited RCU so I could be missing the point.
Yes, expedited grace periods use the common sequence-number functions. Oddly enough, normal grace periods were the last to make use of these.
Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
kernel/rcu/rcuperf.c +48
-0+else + b_rcu_gp_test_finished = cur_ops->get_gp_seq (); + +pr_alert ("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n" , + (unsigned long long )(end_time - start_time), kfree_loops, +rcuperf_seq_diff (b_rcu_gp_test_finished, b_rcu_gp_test_started)); +if (shutdown) { +smp_mb ();/* Assign before wake. */ +wake_up (&shutdown_wq); + } + } + +torture_kthread_stopping ("kfree_perf_thread" ); +return 0 ; +} + +static void +kfree_perf_cleanup (void ) +{ +int i; + +if (torture_cleanup_begin ()) +return ; + +if (kfree_reader_tasks) { +for (i =0 ; i < kfree_nrealthreads; i++) +torture_stop_kthread (kfree_perf_thread, + kfree_reader_tasks[i]); +kfree (kfree_reader_tasks); + } + +torture_cleanup_end (); +} + +/* +* shutdown kthread. Just waits to be awakened, then shuts down system. +*/ +static int +kfree_perf_shutdown (void *arg) +{ +do { +wait_event (shutdown_wq, +atomic_read (&n_kfree_perf_thread_ended) >= + kfree_nrealthreads); + }while (atomic_read (&n_kfree_perf_thread_ended) < kfree_nrealthreads); + +smp_mb ();/* Wake before output. */ +Quoted 8 lines
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 703
Quoted 3 lines
kernel/rcu/rcuperf.c +3
-0+kfree_perf_cleanup (); +kernel_power_off (); +return -EINVAL ;These last four lines should be combined with those of
rcu_perf_shutdown(). Actually, you could fold the two functions together with only a pair of arguments and two one-line wrapper functions, which would be even better.
Quoted 3 lines
But the
cleanup()function is different in the 2 cases and will have to be passed in as a function pointer. I believe we discussed this last review as well.
Calling through a pointer should be a non-problem in this case. We are nowhere near a fastpath.
Thanx, Paul
Quoted 37 lines · Paul E. McKenney
Quoted 1 line · Joel Fernandes
On Wed, Aug 28, 2019 at 02:12:26PM -0700, Paul E. McKenney wrote:
[ . . . ]
Quoted 32 lines
Quoted 14 lines
Quoted 14 lines
kernel/rcu/rcuperf.c +14
-0+static int +kfree_perf_thread (void *arg) +{ +int i, loop =0 ; +long me = (long )arg; +struct kfree_obj *alloc_ptr; +u64 start_time, end_time; + +VERBOSE_PERFOUT_STRING ("kfree_perf_thread task started" ); +set_cpus_allowed_ptr (current,cpumask_of (me % nr_cpu_ids)); +set_user_nice (current,MAX_NICE ); + + start_time =ktime_get_mono_fast_ns (); +Quoted 8 lines
kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 624
Quoted 3 lines
kernel/rcu/rcuperf.c +3
-0+if (atomic_inc_return (&n_kfree_perf_thread_started) >= kfree_nrealthreads) { +if (gp_exp) + b_rcu_gp_test_started = cur_ops->exp_completed () /2 ;At some point, it would be good to use the new grace-period sequence-counter functions (
rcuperf_seq_diff(), for example) instead of the open-coded division by 2. I freely admit that you are just copying my obsolete hack in this case, so not needed in this patch.But I am using
rcu_seq_diff()below in thepr_alert().Anyway, I agree this can be a follow-on since this pattern is borrowed from another part of rcuperf. However, I am also confused about the pattern itself.
If I understand, you are doing the "/ 2" because
expedited_sequenceprogresses by 2 for every expedited batch.
Quoted 7 lines
Quoted 4 lines
But does
rcu_seq_diff()really work on these expedited GP numbers, and will it be immune to changes inRCU_SEQ_STATE_MASK? Sorry for the silly questions, but admittedly I have not looked too much yet into expedited RCU so I could be missing the point.Yes, expedited grace periods use the common sequence-number functions. Oddly enough, normal grace periods were the last to make use of these.
Ok, will clean up in a follow on patch as we agreed, so as to not block this series.
Quoted 57 lines
Quoted 57 lines
Quoted 56 lines
Quoted 41 lines
kernel/rcu/rcuperf.c +41
-0+else + b_rcu_gp_test_finished = cur_ops->get_gp_seq (); + +pr_alert ("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n" , + (unsigned long long )(end_time - start_time), kfree_loops, +rcuperf_seq_diff (b_rcu_gp_test_finished, b_rcu_gp_test_started)); +if (shutdown) { +smp_mb ();/* Assign before wake. */ +wake_up (&shutdown_wq); + } + } + +torture_kthread_stopping ("kfree_perf_thread" ); +return 0 ; +} + +static void +kfree_perf_cleanup (void ) +{ +int i; + +if (torture_cleanup_begin ()) +return ; + +if (kfree_reader_tasks) { +for (i =0 ; i < kfree_nrealthreads; i++) +torture_stop_kthread (kfree_perf_thread, + kfree_reader_tasks[i]); +kfree (kfree_reader_tasks); + } + +torture_cleanup_end (); +} + +/* +* shutdown kthread. Just waits to be awakened, then shuts down system. +*/ +static int +kfree_perf_shutdown (void *arg) +{ + do {kernel/rcu/rcuperf.c rcu_perf_shutdown(void *arg) · line 703
Quoted 10 lines
kernel/rcu/rcuperf.c +10
-0+wait_event (shutdown_wq, +atomic_read (&n_kfree_perf_thread_ended) >= + kfree_nrealthreads); + }while (atomic_read (&n_kfree_perf_thread_ended) < kfree_nrealthreads); + +smp_mb ();/* Wake before output. */ + +kfree_perf_cleanup (); +kernel_power_off (); +return -EINVAL ;These last four lines should be combined with those of
rcu_perf_shutdown(). Actually, you could fold the two functions together with only a pair of arguments and two one-line wrapper functions, which would be even better.
Quoted 6 lines
Quoted 3 lines
But the
cleanup()function is different in the 2 cases and will have to be passed in as a function pointer. I believe we discussed this last review as well.Calling through a pointer should be a non-problem in this case. We are nowhere near a fastpath.
There's also the wait_event() condition that is different. I don't see how we can combine this. It will look though and with probably the same lines of code. Can this function be as is? pretty-please :-D. Or perhaps, if you feel it is a trivial cleanup, could you do it so I can understand what you mean? Sorry!
thanks!
- Joel
PATCH 2/5 rcu/tree: Add multiple in-flight batches of kfree_rcu work
In reply to a message not in the archiveDuring testing, it was observed that amount of memory consumed due kfree_rcu() batching is 300-400MB. Previously we had only a single head_free pointer pointing to the list of rcu_head(s) that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g the head_free list.
We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
--- kernel/rcu/tree.c | 64 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 19 deletions(-)
kernel/rcu/tree.c +45 -19
@@ -2688,28 +2688,38 @@ EXPORT_SYMBOL_GPL(call_rcu);/* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) +#define KFREE_N_BATCHES 2 + +struct kfree_rcu_work { +/* The rcu_work node for queuing work with queue_rcu_work(). The work +* is done after a grace period. +*/ +struct rcu_work rcu_work; + +/* The list of objects that have now left ->head and are queued for +* freeing after a grace period. +*/ +struct rcu_head *head_free; + +struct kfree_rcu_cpu *krcp; +}; +static DEFINE_PER_CPU(__typeof__(struct kfree_rcu_work)[KFREE_N_BATCHES], krw);/* * Maximum number of kfree(s) to batch, if this limit is hit then the batch of * kfree(s) is queued for freeing after a grace period, right away. */ struct kfree_rcu_cpu { -/* The rcu_work node for queuing work with queue_rcu_work(). The work -* is done after a grace period. -*/ -struct rcu_work rcu_work;/* The list of objects being queued in a batch but are not yet * scheduled to be freed. */ struct rcu_head *head; -/* The list of objects that have now left ->head and are queued for -* freeing after a grace period. -*/ -struct rcu_head *head_free; +/* Pointer to the per-cpu array of kfree_rcu_work structures */ +struct kfree_rcu_work *krwp; -/* Protect concurrent access to this structure. */ +/* Protect concurrent access to this structure and kfree_rcu_work. */ spinlock_t lock;/* The delayed work that flushes ->head to ->head_free incase ->head
@@ -2730,12 +2740,14 @@ static void kfree_rcu_work(struct work_struct *work) {unsigned long flags;struct rcu_head *head, *next; -struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), - struct kfree_rcu_cpu, rcu_work); +struct kfree_rcu_work *krwp =container_of (to_rcu_work (work), + struct kfree_rcu_work, rcu_work); +struct kfree_rcu_cpu *krcp; + + krcp = krwp->krcp;spin_lock_irqsave (&krcp->lock, flags); - head = krcp->head_free; - krcp->head_free =NULL ; + head =xchg (&krwp->head_free,NULL );spin_unlock_irqrestore (&krcp->lock, flags);/*
@@ -2758,19 +2770,28 @@ static void kfree_rcu_work(struct work_struct *work)*/ static inline bool queue_kfree_rcu_work (struct kfree_rcu_cpu *krcp) { +int i =0 ; +struct kfree_rcu_work *krwp =NULL ; +lockdep_assert_held (&krcp->lock); +while (i <KFREE_N_BATCHES ) { +if (!krcp->krwp[i].head_free) { + krwp = &(krcp->krwp[i]); +break ; + } + i++; + } -/* If a previous RCU batch work is already in progress, we cannot queue +/* If both RCU batches are already in progress, we cannot queue * another one, just refuse the optimization and it will be retried * again in KFREE_DRAIN_JIFFIES time. */ -if (krcp->head_free) +if (!krwp)return false; - krcp->head_free = krcp->head; - krcp->head =NULL ; -INIT_RCU_WORK (&krcp->rcu_work, kfree_rcu_work); -queue_rcu_work (system_wq, &krcp->rcu_work); + krwp->head_free =xchg (&krcp->head,NULL ); +INIT_RCU_WORK (&krwp->rcu_work, kfree_rcu_work); +queue_rcu_work (system_wq, &krwp->rcu_work);return true; }
@@ -3736,8 +3757,13 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); +struct kfree_rcu_work *krwp = &(per_cpu (krw, cpu)[0 ]); +int i =KFREE_N_BATCHES ;spin_lock_init (&krcp->lock); + krcp->krwp = krwp; +while (i--) + krwp[i].krcp = krcp;INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); } }
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH 2/5] rcu/tree: Add multiple in-flight batches of kfree_rcu work
Hi Joel,
Quoted 34 lines · Joel Fernandes (Google)
During testing, it was observed that amount of memory consumed due
kfree_rcu()batching is 300-400MB. Previously we had only a singlehead_freepointer pointing to the list ofrcu_head(s)that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g thehead_freelist.We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
Suggested-by Paul E. McKenney <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/rcu/tree.c | 64 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 19 deletions(-)kernel/rcu/tree.c +7
-0@@ -2688,28 +2688,38 @@ EXPORT_SYMBOL_GPL(call_rcu);/* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) +#define KFREE_N_BATCHES 2 + +struct kfree_rcu_work { +/* The rcu_work node for queuing work with queue_rcu_work(). The work +* is done after a grace period. +*/ +struct rcu_work rcu_work;
kernel/rcu/tree.c EXPORT_SYMBOL_GPL(call_rcu); · line 2707
Quoted 10 lines
kernel/rcu/tree.c +9
-0+ +/* The list of objects that have now left ->head and are queued for +* freeing after a grace period. +*/ +struct rcu_head *head_free; + +struct kfree_rcu_cpu *krcp; +}; +static DEFINE_PER_CPU(__typeof__(struct kfree_rcu_work)[KFREE_N_BATCHES], krw);
Why not
static DEFINE_PER_CPU(struct kfree_rcu_work[KFREE_N_BATCHES], krw);
here? Am I missing something?
Further, given "struct kfree_rcu_cpu" is only for defining percpu variables, how about orginazing the data structure like:
struct kfree_rcu_cpu {
...
struct kfree_rcu_work krws[KFREE_N_BATCHES];
...
}This could save one pointer in kfree_rcu_cpu, and I think it provides better cache locality for accessing _cpu and _work on the same cpu.
Thoughts?
Regards,
Boqun
Quoted 98 lines
kernel/rcu/tree.c +29
-19/* * Maximum number of kfree(s) to batch, if this limit is hit then the batch of * kfree(s) is queued for freeing after a grace period, right away. */ struct kfree_rcu_cpu { -/* The rcu_work node for queuing work with queue_rcu_work(). The work -* is done after a grace period. -*/ -struct rcu_work rcu_work;/* The list of objects being queued in a batch but are not yet * scheduled to be freed. */ struct rcu_head *head; -/* The list of objects that have now left ->head and are queued for -* freeing after a grace period. -*/ -struct rcu_head *head_free; +/* Pointer to the per-cpu array of kfree_rcu_work structures */ +struct kfree_rcu_work *krwp; -/* Protect concurrent access to this structure. */ +/* Protect concurrent access to this structure and kfree_rcu_work. */ spinlock_t lock;/* The delayed work that flushes ->head to ->head_free incase ->head @@ -2730,12 +2740,14 @@ static void kfree_rcu_work(struct work_struct *work) {unsigned long flags;struct rcu_head *head, *next; -struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), - struct kfree_rcu_cpu, rcu_work); +struct kfree_rcu_work *krwp =container_of (to_rcu_work (work), + struct kfree_rcu_work, rcu_work); +struct kfree_rcu_cpu *krcp; + + krcp = krwp->krcp;spin_lock_irqsave (&krcp->lock, flags); - head = krcp->head_free; - krcp->head_free =NULL ; + head =xchg (&krwp->head_free,NULL );spin_unlock_irqrestore (&krcp->lock, flags);/* @@ -2758,19 +2770,28 @@ static void kfree_rcu_work(struct work_struct *work)*/ static inline bool queue_kfree_rcu_work (struct kfree_rcu_cpu *krcp) { +int i =0 ; +struct kfree_rcu_work *krwp =NULL ; +lockdep_assert_held (&krcp->lock); +while (i <KFREE_N_BATCHES ) { +if (!krcp->krwp[i].head_free) { + krwp = &(krcp->krwp[i]); +break ; + } + i++; + } -/* If a previous RCU batch work is already in progress, we cannot queue +/* If both RCU batches are already in progress, we cannot queue * another one, just refuse the optimization and it will be retried * again in KFREE_DRAIN_JIFFIES time. */ -if (krcp->head_free) +if (!krwp)return false; - krcp->head_free = krcp->head; - krcp->head =NULL ; -INIT_RCU_WORK (&krcp->rcu_work, kfree_rcu_work); -queue_rcu_work (system_wq, &krcp->rcu_work); + krwp->head_free =xchg (&krcp->head,NULL ); +INIT_RCU_WORK (&krwp->rcu_work, kfree_rcu_work); +queue_rcu_work (system_wq, &krwp->rcu_work);return true; }@@ -3736,8 +3757,13 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); +struct kfree_rcu_work *krwp = &(per_cpu (krw, cpu)[0 ]); +int i =KFREE_N_BATCHES ;spin_lock_init (&krcp->lock); + krcp->krwp = krwp; +while (i--) + krwp[i].krcp = krcp;INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); } }Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH 2/5] rcu/tree: Add multiple in-flight batches of kfree_rcu work
Quoted 50 lines · Boqun Feng
Hi Joel,
Quoted 34 lines · Joel Fernandes (Google)
During testing, it was observed that amount of memory consumed due
kfree_rcu()batching is 300-400MB. Previously we had only a singlehead_freepointer pointing to the list ofrcu_head(s)that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g thehead_freelist.We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
Suggested-by Paul E. McKenney <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/rcu/tree.c | 64 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 19 deletions(-)kernel/rcu/tree.c +7
-0@@ -2688,28 +2688,38 @@ EXPORT_SYMBOL_GPL(call_rcu);/* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) +#define KFREE_N_BATCHES 2 + +struct kfree_rcu_work { +/* The rcu_work node for queuing work with queue_rcu_work(). The work +* is done after a grace period. +*/ +struct rcu_work rcu_work;kernel/rcu/tree.c EXPORT_SYMBOL_GPL(call_rcu); · line 2707
Quoted 10 lines
kernel/rcu/tree.c +9
-0+ +/* The list of objects that have now left ->head and are queued for +* freeing after a grace period. +*/ +struct rcu_head *head_free; + +struct kfree_rcu_cpu *krcp; +}; +static DEFINE_PER_CPU(__typeof__(struct kfree_rcu_work)[KFREE_N_BATCHES], krw);Why not
Quoted 3 lines
static DEFINE_PER_CPU(struct kfree_rcu_work[KFREE_N_BATCHES], krw);here? Am I missing something?
Yes, that's better.
Quoted 9 lines
Further, given "struct
kfree_rcu_cpu" is only for defining percpu variables, how about orginazing the data structure like:struct kfree_rcu_cpu { ... struct kfree_rcu_work krws[KFREE_N_BATCHES]; ... }
Quoted 4 lines
This could save one pointer in
kfree_rcu_cpu, and I think it provides better cache locality for accessing_cpuand_workon the same cpu.Thoughts?
Yes, that's better. Thanks, Boqun! Following is the diff which I will fold into this patch:
---8<-----------------------
kernel/rcu/tree.c +4 -7
@@ -2717,7 +2717,6 @@ struct kfree_rcu_work {struct kfree_rcu_cpu *krcp; }; -static DEFINE_PER_CPU(__typeof__(struct kfree_rcu_work)[KFREE_N_BATCHES], krw);/* * Maximum number of kfree(s) to batch, if this limit is hit then the batch of
@@ -2731,7 +2730,7 @@ struct kfree_rcu_cpu {struct rcu_head *head;/* Pointer to the per-cpu array of kfree_rcu_work structures */ -struct kfree_rcu_work *krwp; +struct kfree_rcu_work krw_arr[KFREE_N_BATCHES ];/* Protect concurrent access to this structure and kfree_rcu_work. */ spinlock_t lock;
@@ -2800,8 +2799,8 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)lockdep_assert_held (&krcp->lock);while (i <KFREE_N_BATCHES ) { -if (!krcp->krwp[i].head_free) { - krwp = &(krcp->krwp[i]); +if (!krcp->krw_arr[i].head_free) { + krwp = &(krcp->krw_arr[i]);break ; } i++;
@@ -3780,13 +3779,11 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); -struct kfree_rcu_work *krwp = &(per_cpu (krw, cpu)[0 ]);int i =KFREE_N_BATCHES ;spin_lock_init (&krcp->lock); - krcp->krwp = krwp;while (i--) - krwp[i].krcp = krcp; + krcp->krw_arr[i].krcp = krcp;INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); } }
PATCH v2 1/1 rcu/tree: Add multiple in-flight batches of kfree_rcu work
In reply to Joel Fernandes (Google)During testing, it was observed that amount of memory consumed due kfree_rcu() batching is 300-400MB. Previously we had only a single head_free pointer pointing to the list of rcu_head(s) that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g the head_free list.
We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
--- kernel/rcu/tree.c | 61 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 19 deletions(-)
kernel/rcu/tree.c +42 -19
@@ -2688,28 +2688,37 @@ EXPORT_SYMBOL_GPL(call_rcu);/* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) +#define KFREE_N_BATCHES 2 + +struct kfree_rcu_work { +/* The rcu_work node for queuing work with queue_rcu_work(). The work +* is done after a grace period. +*/ +struct rcu_work rcu_work; + +/* The list of objects that have now left ->head and are queued for +* freeing after a grace period. +*/ +struct rcu_head *head_free; + +struct kfree_rcu_cpu *krcp; +};/* * Maximum number of kfree(s) to batch, if this limit is hit then the batch of * kfree(s) is queued for freeing after a grace period, right away. */ struct kfree_rcu_cpu { -/* The rcu_work node for queuing work with queue_rcu_work(). The work -* is done after a grace period. -*/ -struct rcu_work rcu_work;/* The list of objects being queued in a batch but are not yet * scheduled to be freed. */ struct rcu_head *head; -/* The list of objects that have now left ->head and are queued for -* freeing after a grace period. -*/ -struct rcu_head *head_free; +/* Pointer to the per-cpu array of kfree_rcu_work structures */ +struct kfree_rcu_work krw_arr[KFREE_N_BATCHES ]; -/* Protect concurrent access to this structure. */ +/* Protect concurrent access to this structure and kfree_rcu_work. */ spinlock_t lock;/* The delayed work that flushes ->head to ->head_free incase ->head
@@ -2730,12 +2739,14 @@ static void kfree_rcu_work(struct work_struct *work) {unsigned long flags;struct rcu_head *head, *next; -struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), - struct kfree_rcu_cpu, rcu_work); +struct kfree_rcu_work *krwp =container_of (to_rcu_work (work), + struct kfree_rcu_work, rcu_work); +struct kfree_rcu_cpu *krcp; + + krcp = krwp->krcp;spin_lock_irqsave (&krcp->lock, flags); - head = krcp->head_free; - krcp->head_free =NULL ; + head =xchg (&krwp->head_free,NULL );spin_unlock_irqrestore (&krcp->lock, flags);/*
@@ -2758,19 +2769,28 @@ static void kfree_rcu_work(struct work_struct *work)*/ static inline bool queue_kfree_rcu_work (struct kfree_rcu_cpu *krcp) { +int i =0 ; +struct kfree_rcu_work *krwp =NULL ; +lockdep_assert_held (&krcp->lock); +while (i <KFREE_N_BATCHES ) { +if (!krcp->krw_arr[i].head_free) { + krwp = &(krcp->krw_arr[i]); +break ; + } + i++; + } -/* If a previous RCU batch work is already in progress, we cannot queue +/* If both RCU batches are already in progress, we cannot queue * another one, just refuse the optimization and it will be retried * again in KFREE_DRAIN_JIFFIES time. */ -if (krcp->head_free) +if (!krwp)return false; - krcp->head_free = krcp->head; - krcp->head =NULL ; -INIT_RCU_WORK (&krcp->rcu_work, kfree_rcu_work); -queue_rcu_work (system_wq, &krcp->rcu_work); + krwp->head_free =xchg (&krcp->head,NULL ); +INIT_RCU_WORK (&krwp->rcu_work, kfree_rcu_work); +queue_rcu_work (system_wq, &krwp->rcu_work);return true; }
@@ -3736,8 +3756,11 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); +int i =KFREE_N_BATCHES ;spin_lock_init (&krcp->lock); +while (i--) + krcp->krw_arr[i].krcp = krcp;INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); } }
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerRe: [PATCH v2] rcu/tree: Add multiple in-flight batches of kfree_rcu work
Quoted 8 lines · Joel Fernandes (Google)
During testing, it was observed that amount of memory consumed due
kfree_rcu()batching is 300-400MB. Previously we had only a singlehead_freepointer pointing to the list ofrcu_head(s)that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g thehead_freelist.
Quoted 4 lines
We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
Nice! A few comments below. Please address them and post a full v3. (I am off the next two days, and I guarantee you that upon return I will mix and match the wrong patches otherwise!)
Quoted 63 lines
Suggested-by Paul E. McKenney <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/rcu/tree.c | 61 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 19 deletions(-)kernel/rcu/tree.c +18
-11@@ -2688,28 +2688,37 @@ EXPORT_SYMBOL_GPL(call_rcu);/* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) +#define KFREE_N_BATCHES 2 + +struct kfree_rcu_work { +/* The rcu_work node for queuing work with queue_rcu_work(). The work +* is done after a grace period. +*/ +struct rcu_work rcu_work; + +/* The list of objects that have now left ->head and are queued for +* freeing after a grace period. +*/ +struct rcu_head *head_free; + +struct kfree_rcu_cpu *krcp; +};/* * Maximum number of kfree(s) to batch, if this limit is hit then the batch of * kfree(s) is queued for freeing after a grace period, right away. */ struct kfree_rcu_cpu { -/* The rcu_work node for queuing work with queue_rcu_work(). The work -* is done after a grace period. -*/ -struct rcu_work rcu_work;/* The list of objects being queued in a batch but are not yet * scheduled to be freed. */ struct rcu_head *head; -/* The list of objects that have now left ->head and are queued for -* freeing after a grace period. -*/ -struct rcu_head *head_free; +/* Pointer to the per-cpu array of kfree_rcu_work structures */ +struct kfree_rcu_work krw_arr[KFREE_N_BATCHES ]; -/* Protect concurrent access to this structure. */ +/* Protect concurrent access to this structure and kfree_rcu_work. */ spinlock_t lock;/* The delayed work that flushes ->head to ->head_free incase ->head @@ -2730,12 +2739,14 @@ static void kfree_rcu_work(struct work_struct *work) {unsigned long flags;struct rcu_head *head, *next; -struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), - struct kfree_rcu_cpu, rcu_work);
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2749
Quoted 10 lines
kernel/rcu/tree.c +6
-2+struct kfree_rcu_work *krwp =container_of (to_rcu_work (work), + struct kfree_rcu_work, rcu_work); +struct kfree_rcu_cpu *krcp; + + krcp = krwp->krcp;spin_lock_irqsave (&krcp->lock, flags); - head = krcp->head_free; - krcp->head_free =NULL ; + head =xchg (&krwp->head_free,NULL );
Given that we hold the lock, why the xchg()? Alternatively, why not just acquire the lock in the other places you use xchg()? This is a per-CPU lock, so contention should not be a problem, should it?
Quoted 14 lines
kernel/rcu/tree.c +6
-0spin_unlock_irqrestore (&krcp->lock, flags);/* @@ -2758,19 +2769,28 @@ static void kfree_rcu_work(struct work_struct *work)*/ static inline bool queue_kfree_rcu_work (struct kfree_rcu_cpu *krcp) { +int i =0 ; +struct kfree_rcu_work *krwp =NULL ; +lockdep_assert_held (&krcp->lock); +while (i <KFREE_N_BATCHES ) { +if (!krcp->krw_arr[i].head_free) { + krwp = &(krcp->krw_arr[i]);
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2787
Quoted 10 lines
kernel/rcu/tree.c +5
-1+break ; + } + i++; + } -/* If a previous RCU batch work is already in progress, we cannot queue +/* If both RCU batches are already in progress, we cannot queue * another one, just refuse the optimization and it will be retried * again in KFREE_DRAIN_JIFFIES time. */
If you are going to remove the traditional first "/*" line of a comment, why not go all the way and cut the last one as well? "//".
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2791
Quoted 9 lines
kernel/rcu/tree.c +2
-5-if (krcp->head_free) +if (!krwp)return false; - krcp->head_free = krcp->head; - krcp->head =NULL ; -INIT_RCU_WORK (&krcp->rcu_work, kfree_rcu_work); -queue_rcu_work (system_wq, &krcp->rcu_work); + krwp->head_free =xchg (&krcp->head,NULL );
This isn't anywhere near a fastpath, so just acquiring the lock is a better choice here.
Quoted 6 lines
kernel/rcu/tree.c +2
-0+INIT_RCU_WORK (&krwp->rcu_work, kfree_rcu_work); +queue_rcu_work (system_wq, &krwp->rcu_work);return true; }
kernel/rcu/tree.c static void __init kfree_rcu_batch_init(void) · line 3763
Quoted 8 lines
kernel/rcu/tree.c +3
-0@@ -3736,8 +3756,11 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); +int i =KFREE_N_BATCHES ;spin_lock_init (&krcp->lock); +while (i--) + krcp->krw_arr[i].krcp = krcp;
This was indeed a nice trick back in the PDP-11 days of 64-kilobyte address spaces, so thank you for the nostalgia! However, a straight-up "for" loop is less vulnerable to code being added between the declaration of "i" and the "while" loop.
Quoted 6 lines
kernel/rcu/tree.c +0
-0INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); } }Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH v2] rcu/tree: Add multiple in-flight batches of kfree_rcu work
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Joel Fernandes (Google)
During testing, it was observed that amount of memory consumed due
kfree_rcu()batching is 300-400MB. Previously we had only a singlehead_freepointer pointing to the list ofrcu_head(s)that are to be freed after a grace period. Until this list is drained, we cannot queue any more objects on it since such objects may not be ready to be reclaimed when the worker thread eventually gets to drainin g thehead_freelist.We can do better by maintaining multiple lists as done by this patch. Testing shows that memory consumption came down by around 100-150MB with just adding another list. Adding more than 1 additional list did not show any improvement.
[snip]
Quoted 12 lines
Quoted 12 lines
kernel/rcu/tree.c +5
-2@@ -2730,12 +2739,14 @@ static void kfree_rcu_work(struct work_struct *work) {unsigned long flags;struct rcu_head *head, *next; -struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), - struct kfree_rcu_cpu, rcu_work); +struct kfree_rcu_work *krwp =container_of (to_rcu_work (work), + struct kfree_rcu_work, rcu_work); +struct kfree_rcu_cpu *krcp; + + krcp = krwp->krcp;
Quoted 8 lines
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2749
Quoted 4 lines
kernel/rcu/tree.c +1
-2spin_lock_irqsave (&krcp->lock, flags); - head = krcp->head_free; - krcp->head_free =NULL ; + head =xchg (&krwp->head_free,NULL );Given that we hold the lock, why the
xchg()? Alternatively, why not just acquire the lock in the other places you usexchg()? This is a per-CPU lock, so contention should not be a problem, should it?
I realized I was being silly :(. Was trying to reduce lines of code and hence implemented it like that as a one-liner. Locking protocol is not needed or intended for that xchg since as pointed, a lock is held.
Quoted 19 lines
Quoted 19 lines
kernel/rcu/tree.c +10
-0spin_unlock_irqrestore (&krcp->lock, flags);/* @@ -2758,19 +2769,28 @@ static void kfree_rcu_work(struct work_struct *work)*/ static inline bool queue_kfree_rcu_work (struct kfree_rcu_cpu *krcp) { +int i =0 ; +struct kfree_rcu_work *krwp =NULL ; +lockdep_assert_held (&krcp->lock); +while (i <KFREE_N_BATCHES ) { +if (!krcp->krw_arr[i].head_free) { + krwp = &(krcp->krw_arr[i]); +break ; + } + i++; + }
Quoted 8 lines
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2787
Quoted 5 lines
kernel/rcu/tree.c +1
-1-/* If a previous RCU batch work is already in progress, we cannot queue +/* If both RCU batches are already in progress, we cannot queue * another one, just refuse the optimization and it will be retried * again in KFREE_DRAIN_JIFFIES time. */ If you are going to remove the traditional first "/*" line of a comment, why not go all the way and cut the last one as well? "//".
Will add the /* in the beginning :)
Quoted 4 lines
Quoted 4 lines
kernel/rcu/tree.c +1
-1-if (krcp->head_free) +if (!krwp)return false;
Quoted 8 lines
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2791
Quoted 5 lines
kernel/rcu/tree.c +1
-4- krcp->head_free = krcp->head; - krcp->head =NULL ; -INIT_RCU_WORK (&krcp->rcu_work, kfree_rcu_work); -queue_rcu_work (system_wq, &krcp->rcu_work); + krwp->head_free =xchg (&krcp->head,NULL );This isn't anywhere near a fastpath, so just acquiring the lock is a better choice here.
My reasoning was same as above. Will change it to 2 statements since lock is already held.
Quoted 11 lines
Quoted 11 lines
kernel/rcu/tree.c +3
-0+INIT_RCU_WORK (&krwp->rcu_work, kfree_rcu_work); +queue_rcu_work (system_wq, &krwp->rcu_work);return true; }@@ -3736,8 +3756,11 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); +int i =KFREE_N_BATCHES ;
Quoted 8 lines
kernel/rcu/tree.c static void __init kfree_rcu_batch_init(void) · line 3763
Quoted 3 lines
kernel/rcu/tree.c +2
-0spin_lock_init (&krcp->lock); +while (i--) + krcp->krw_arr[i].krcp = krcp;This was indeed a nice trick back in the PDP-11 days of 64-kilobyte address spaces, so thank you for the nostalgia! However, a straight-up "for" loop is less vulnerable to code being added between the declaration of "i" and the "while" loop.
Ok, will do.
thanks,
- Joel
PATCH 3/5 rcu/tree: Add support for debug_objects debugging for kfree_rcu()
In reply to a message not in the archiveMake use of RCU's debug_objects debugging support (CONFIG_DEBUG_OBJECTS_RCU_HEAD) similar to call_rcu() and other flavors. We queue the object during the kfree_rcu() call and dequeue it during reclaim.
Tested that enabling CONFIG_DEBUG_OBJECTS_RCU_HEAD successfully detects double kfree_rcu() calls.
--- kernel/rcu/tree.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/rcu/tree.c +8 -0
@@ -2757,6 +2757,7 @@ static void kfree_rcu_work(struct work_struct *work)for (; head; head = next) { next = head->next;/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head);__rcu_reclaim (rcu_state.name, head);cond_resched_tasks_rcu_qs (); }
@@ -2868,6 +2869,13 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING )return kfree_call_rcu_nobatch (head, func); +if (debug_rcu_head_queue (head)) { +/* Probable double kfree_rcu() */ +WARN_ONCE (1 ,"kfree_call_rcu(): Double-freed call. rcu_head %p\n" , + head); +return ; + } + head->func = func;local_irq_save (flags);/* For safely calling this_cpu_ptr(). */
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerRe: [PATCH 3/5] rcu/tree: Add support for debug_objects debugging for kfree_rcu()
Quoted 2 lines · Joel Fernandes (Google)
Make use of RCU's
debug_objectsdebugging support (CONFIG_DEBUG_OBJECTS_RCU_HEAD) similar tocall_rcu()and other flavors.
Other flavors? Ah, call_srcu(), rcu_barrier(), and srcu_barrier(), right?
Quoted 7 lines
We queue the object during the
kfree_rcu()call and dequeue it during reclaim.Tested that enabling
CONFIG_DEBUG_OBJECTS_RCU_HEADsuccessfully detects doublekfree_rcu()calls.Signed-off-by Joel Fernandes (Google) <[email protected]>
The code looks good!
Thanx, Paul
Quoted 33 lines
--- kernel/rcu/tree.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/rcu/tree.c +8
-0@@ -2757,6 +2757,7 @@ static void kfree_rcu_work(struct work_struct *work)for (; head; head = next) { next = head->next;/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head);__rcu_reclaim (rcu_state.name, head);cond_resched_tasks_rcu_qs (); }@@ -2868,6 +2869,13 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING )return kfree_call_rcu_nobatch (head, func); +if (debug_rcu_head_queue (head)) { +/* Probable double kfree_rcu() */ +WARN_ONCE (1 ,"kfree_call_rcu(): Double-freed call. rcu_head %p\n" , + head); +return ; + } + head->func = func;local_irq_save (flags);/* For safely calling this_cpu_ptr(). */ Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH 3/5] rcu/tree: Add support for debug_objects debugging for kfree_rcu()
Quoted 6 lines · Paul E. McKenney
Quoted 2 lines · Joel Fernandes (Google)
Make use of RCU's
debug_objectsdebugging support (CONFIG_DEBUG_OBJECTS_RCU_HEAD) similar tocall_rcu()and other flavors.Other flavors? Ah,
call_srcu(),rcu_barrier(), andsrcu_barrier(), right?
Yes.
Quoted 9 lines
Quoted 7 lines
We queue the object during the
kfree_rcu()call and dequeue it during reclaim.Tested that enabling
CONFIG_DEBUG_OBJECTS_RCU_HEADsuccessfully detects doublekfree_rcu()calls.Signed-off-by Joel Fernandes (Google) <[email protected]>The code looks good!
thanks, does that mean you'll ack/apply it? :-P
- Joel
Quoted 36 lines
Thanx, Paul
Quoted 33 lines
--- kernel/rcu/tree.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/rcu/tree.c +8
-0@@ -2757,6 +2757,7 @@ static void kfree_rcu_work(struct work_struct *work)for (; head; head = next) { next = head->next;/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head);__rcu_reclaim (rcu_state.name, head);cond_resched_tasks_rcu_qs (); }@@ -2868,6 +2869,13 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING )return kfree_call_rcu_nobatch (head, func); +if (debug_rcu_head_queue (head)) { +/* Probable double kfree_rcu() */ +WARN_ONCE (1 ,"kfree_call_rcu(): Double-freed call. rcu_head %p\n" , + head); +return ; + } + head->func = func;local_irq_save (flags);/* For safely calling this_cpu_ptr(). */ Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerRe: [PATCH 3/5] rcu/tree: Add support for debug_objects debugging for kfree_rcu()
Quoted 13 lines · Joel Fernandes
Quoted 6 lines · Paul E. McKenney
Quoted 2 lines · Joel Fernandes (Google)
Make use of RCU's
debug_objectsdebugging support (CONFIG_DEBUG_OBJECTS_RCU_HEAD) similar tocall_rcu()and other flavors.Other flavors? Ah,
call_srcu(),rcu_barrier(), andsrcu_barrier(), right?Yes.
Quoted 3 lines
Quoted 3 lines
We queue the object during the
kfree_rcu()call and dequeue it during reclaim.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Tested that enabling
CONFIG_DEBUG_OBJECTS_RCU_HEADsuccessfully detects doublekfree_rcu()calls.Signed-off-by Joel Fernandes (Google) <[email protected]>The code looks good!
thanks, does that mean you'll ack/apply it? :-P
Is it independent of 1/5 and 2/5?
Thanx, Paul
Quoted 38 lines
- Joel
Quoted 36 lines
Thanx, Paul
Quoted 33 lines
--- kernel/rcu/tree.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/rcu/tree.c +8
-0@@ -2757,6 +2757,7 @@ static void kfree_rcu_work(struct work_struct *work)for (; head; head = next) { next = head->next;/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head);__rcu_reclaim (rcu_state.name, head);cond_resched_tasks_rcu_qs (); }@@ -2868,6 +2869,13 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING )return kfree_call_rcu_nobatch (head, func); +if (debug_rcu_head_queue (head)) { +/* Probable double kfree_rcu() */ +WARN_ONCE (1 ,"kfree_call_rcu(): Double-freed call. rcu_head %p\n" , + head); +return ; + } + head->func = func;local_irq_save (flags);/* For safely calling this_cpu_ptr(). */ Signature · 1 line
2.23.0.187.g17f5b7556c-goog
PATCH 4/5 rcu: Remove kfree_rcu() special casing and lazy handling
In reply to a message not in the archiveRemove kfree_rcu() special casing and lazy handling from RCU. For Tiny RCU we fold the special handling into just Tiny RCU code.
--- Documentation/RCU/stallwarn.txt | 13 +++------ include/linux/rcu_segcblist.h | 2 -- include/trace/events/rcu.h | 32 +++++++++------------- kernel/rcu/rcu.h | 27 ------------------- kernel/rcu/rcu_segcblist.c | 25 +++-------------- kernel/rcu/rcu_segcblist.h | 25 ++--------------- kernel/rcu/srcutree.c | 4 +-- kernel/rcu/tiny.c | 29 +++++++++++++++++++- kernel/rcu/tree.c | 48 ++++++++++++++++++++++++--------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 42 +++++++---------------------- kernel/rcu/tree_stall.h | 6 ++--- 12 files changed, 97 insertions(+), 157 deletions(-)
Documentation/RCU/stallwarn.txt +4 -9
@@ -225,18 +225,12 @@ an estimate of the total number of RCU callbacks queued across all CPUs In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed for each CPU: - 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D + 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1 The "last_accelerate:" prints the low-order 16 bits (in hex) of the jiffies counter when this CPU last invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from -rcu_prepare_for_idle(). The "Nonlazy posted:" indicates lazy-callback -status, so that an "l" indicates that all callbacks were lazy at the start -of the last idle period and an "L" indicates that there are currently -no non-lazy callbacks (in both cases, "." is printed otherwise, as -shown above) and "D" indicates that dyntick-idle processing is enabled -("." is printed otherwise, for example, if disabled via the "nohz=" -kernel boot parameter). +rcu_prepare_for_idle(). If the grace period ends just as the stall warning starts printing, there will be a spurious stall-warning message, which will include
@@ -249,7 +243,8 @@ possible for a zero-jiffy stall to be flagged in this case, depending on how the stall warning and the grace-period initialization happen to interact. Please note that it is not possible to entirely eliminate this sort of false positive without resorting to things like stop_machine(), -which is overkill for this sort of problem. +which is overkill for this sort of problem. "dyntick_enabled: 1" indicates that +dyntick-idle processing is enabled. If all CPUs and tasks have passed through quiescent states, but the grace period has nevertheless failed to end, the stall-warning splat
include/linux/rcu_segcblist.h +0 -2
@@ -22,7 +22,6 @@ struct rcu_cblist {struct rcu_head *head;struct rcu_head **tail;long len; -long len_lazy; };#define RCU_CBLIST_INITIALIZER (n) { .head = NULL, .tail = &n.head }
@@ -73,7 +72,6 @@ struct rcu_segcblist {#else long len;#endif -long len_lazy;u8 enabled;u8 offloaded; };
include/trace/events/rcu.h +12 -20
@@ -474,16 +474,14 @@ TRACE_EVENT_RCU(rcu_dyntick,*/ TRACE_EVENT_RCU(rcu_callback, - TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, - long qlen), + TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen), - TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(void *, func) - __field(long, qlen_lazy) __field(long, qlen) ),
@@ -491,13 +489,12 @@ TRACE_EVENT_RCU(rcu_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->func = rhp->func; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ps %ld/%ld" , +TP_printk ("%s rhp=%p func=%ps %ld" , __entry->rcuname, __entry->rhp, __entry->func, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/*
@@ -511,15 +508,14 @@ TRACE_EVENT_RCU(rcu_callback, TRACE_EVENT_RCU(rcu_kfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, - long qlen_lazy, long qlen), + long qlen), - TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, offset, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(unsigned long, offset) - __field(long, qlen_lazy) __field(long, qlen) ),
@@ -527,13 +523,12 @@ TRACE_EVENT_RCU(rcu_kfree_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->offset = offset; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ld %ld/%ld" , +TP_printk ("%s rhp=%p func=%ld %ld" , __entry->rcuname, __entry->rhp, __entry->offset, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/*
@@ -545,27 +540,24 @@ TRACE_EVENT_RCU(rcu_kfree_callback,*/ TRACE_EVENT_RCU (rcu_batch_start, - TP_PROTO(const char *rcuname, long qlen_lazy, long qlen, long blimit), + TP_PROTO(const char *rcuname, long qlen, long blimit), -TP_ARGS (rcuname, qlen_lazy, qlen, blimit), +TP_ARGS (rcuname, qlen, blimit),TP_STRUCT__entry ( __field(const char *, rcuname) - __field(long, qlen_lazy) __field(long, qlen)__field (long, blimit) ),TP_fast_assign ( __entry->rcuname = rcuname; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; __entry->blimit = blimit; ), -TP_printk ("%s CBs=%ld/%ld bl=%ld" , - __entry->rcuname, __entry->qlen_lazy, __entry->qlen, - __entry->blimit) +TP_printk ("%s CBs=%ld bl=%ld" , + __entry->rcuname, __entry->qlen, __entry->blimit) );/*
kernel/rcu/rcu.h +0 -27
@@ -198,33 +198,6 @@ static inline void debug_rcu_head_unqueue(struct rcu_head *head) }#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */ -void kfree (const void *); - -/* -* Reclaim the specified callback, either by invoking it (non-lazy case) -* or freeing it directly (lazy case). Return true if lazy, false otherwise. -*/ -static inline bool __rcu_reclaim (const char *rn,struct rcu_head *head) -{ -rcu_callback_t f; -unsigned long offset = (unsigned long )head->func; - -rcu_lock_acquire (&rcu_callback_map); -if (__is_kfree_rcu_offset (offset)) { -trace_rcu_invoke_kfree_callback (rn, head, offset); -kfree ((void *)head - offset); -rcu_lock_release (&rcu_callback_map); -return true; - }else { -trace_rcu_invoke_callback (rn, head); - f = head->func; -WRITE_ONCE (head->func, (rcu_callback_t )0L ); -f (head); -rcu_lock_release (&rcu_callback_map); -return false; - } -} -#ifdef CONFIG_RCU_STALL_COMMON extern int rcu_cpu_stall_ftrace_dump;
kernel/rcu/rcu_segcblist.c +3 -22
@@ -20,14 +20,10 @@ void rcu_cblist_init(struct rcu_cblist *rclp) rclp->head =NULL ; rclp->tail = &rclp->head; rclp->len =0 ; - rclp->len_lazy =0 ; }/* * Enqueue an rcu_head structure onto the specified callback list. -* This function assumes that the callback is non-lazy because it -* is intended for use by no-CBs CPUs, which do not distinguish -* between lazy and non-lazy RCU callbacks. */ void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp) {
@@ -54,7 +50,6 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,else drclp->tail = &drclp->head; drclp->len = srclp->len; - drclp->len_lazy = srclp->len_lazy;if (!rhp) {rcu_cblist_init (srclp); }else {
@@ -62,16 +57,12 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, srclp->head = rhp; srclp->tail = &rhp->next;WRITE_ONCE (srclp->len,1 ); - srclp->len_lazy =0 ; } }/* * Dequeue the oldest rcu_head structure from the specified callback -* list. This function assumes that the callback is non-lazy, but -* the caller can later invoke rcu_cblist_dequeued_lazy() if it -* finds otherwise (and if it cares about laziness). This allows -* different users to have different ways of determining laziness. +* list. */ struct rcu_head *rcu_cblist_dequeue (struct rcu_cblist *rclp) {
@@ -161,7 +152,6 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)for (i =0 ; i <RCU_CBLIST_NSEGS ; i++) rsclp->tails[i] = &rsclp->head;rcu_segcblist_set_len (rsclp,0 ); - rsclp->len_lazy =0 ; rsclp->enabled =1 ; }
@@ -173,7 +163,6 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) {WARN_ON_ONCE (!rcu_segcblist_empty (rsclp));WARN_ON_ONCE (rcu_segcblist_n_cbs (rsclp)); -WARN_ON_ONCE (rcu_segcblist_n_lazy_cbs (rsclp)); rsclp->enabled =0 ; }
@@ -253,11 +242,9 @@ bool rcu_segcblist_nextgp(struct rcu_segcblist *rsclp, unsigned long *lp)* absolutely not OK for it to ever miss posting a callback. */ void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is enqueued. */ rhp->next =NULL ;WRITE_ONCE (*rsclp->tails[RCU_NEXT_TAIL ], rhp);
@@ -275,15 +262,13 @@ void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,* period. You have been warned. */ bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {int i;if (rcu_segcblist_n_cbs (rsclp) ==0 )return false;rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is entrained. */ rhp->next =NULL ;for (i =RCU_NEXT_TAIL ; i >RCU_DONE_TAIL ; i--)
@@ -307,8 +292,6 @@ bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rclp->len_lazy += rsclp->len_lazy; - rsclp->len_lazy =0 ; rclp->len =rcu_segcblist_xchg_len (rsclp,0 ); }
@@ -361,9 +344,7 @@ void rcu_segcblist_extract_pend_cbs(struct rcu_segcblist *rsclp,void rcu_segcblist_insert_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rsclp->len_lazy += rclp->len_lazy;rcu_segcblist_add_len (rsclp, rclp->len); - rclp->len_lazy =0 ; rclp->len =0 ; }
kernel/rcu/rcu_segcblist.h +2 -23
@@ -15,15 +15,6 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp)return READ_ONCE (rclp->len); } -/* -* Account for the fact that a previously dequeued callback turned out -* to be marked as lazy. -*/ -static inline void rcu_cblist_dequeued_lazy (struct rcu_cblist *rclp) -{ - rclp->len_lazy--; -} -void rcu_cblist_init (struct rcu_cblist *rclp);void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp); void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,
@@ -59,18 +50,6 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)#endif } -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_lazy_cbs (struct rcu_segcblist *rsclp) -{ -return rsclp->len_lazy; -} - -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_nonlazy_cbs (struct rcu_segcblist *rsclp) -{ -return rcu_segcblist_n_cbs (rsclp) - rsclp->len_lazy; -} -/* * Is the specified rcu_segcblist enabled, for example, not corresponding * to an offline CPU?
@@ -106,9 +85,9 @@ struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_pend_cb (struct rcu_segcblist *rsclp);bool rcu_segcblist_nextgp (struct rcu_segcblist *rsclp,unsigned long *lp);void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp); void rcu_segcblist_extract_done_cbs(struct rcu_segcblist *rsclp,
kernel/rcu/srcutree.c +2 -2
@@ -853,7 +853,7 @@ static void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,local_irq_save (flags); sdp =this_cpu_ptr (ssp->sda);spin_lock_rcu_node (sdp); -rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp, false); +rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp);rcu_segcblist_advance (&sdp->srcu_cblist,rcu_seq_current (&ssp->srcu_gp_seq)); s =rcu_seq_snap (&ssp->srcu_gp_seq);
@@ -1052,7 +1052,7 @@ void srcu_barrier(struct srcu_struct *ssp) sdp->srcu_barrier_head.func = srcu_barrier_cb;debug_rcu_head_queue (&sdp->srcu_barrier_head);if (!rcu_segcblist_entrain (&sdp->srcu_cblist, - &sdp->srcu_barrier_head,0 )) { + &sdp->srcu_barrier_head)) {debug_rcu_head_unqueue (&sdp->srcu_barrier_head);atomic_dec (&ssp->srcu_barrier_cpu_cnt); }
kernel/rcu/tiny.c +28 -1
@@ -73,6 +73,33 @@ void rcu_sched_clock_irq(int user) } } +void kfree (const void *); + +/* +* Reclaim the specified callback, either by invoking it for non-kfree cases or +* freeing it directly (for kfree). Return true if kfreeing, false otherwise. +*/ +static inline bool rcu_reclaim_tiny (struct rcu_head *head) +{ +rcu_callback_t f; +unsigned long offset = (unsigned long )head->func; + +rcu_lock_acquire (&rcu_callback_map); +if (__is_kfree_rcu_offset (offset)) { +trace_rcu_invoke_kfree_callback ("" , head, offset); +kfree ((void *)head - offset); +rcu_lock_release (&rcu_callback_map); +return true; + } + +trace_rcu_invoke_callback ("" , head); + f = head->func; +WRITE_ONCE (head->func, (rcu_callback_t )0L ); +f (head); +rcu_lock_release (&rcu_callback_map); +return false; +} +/* Invoke the RCU callbacks whose grace period has elapsed. */ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused) {
@@ -100,7 +127,7 @@ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unusedprefetch (next);debug_rcu_head_unqueue (list);local_bh_disable (); -__rcu_reclaim ("" , list); +rcu_reclaim_tiny (list);local_bh_enable (); list = next; }
kernel/rcu/tree.c +35 -13
119 lines not on this page · raw message
kernel/rcu/tree.h +0 -1
@@ -183,7 +183,6 @@ struct rcu_data {bool rcu_urgent_qs;/* GP old need light quiescent state. */ bool rcu_forced_tick;/* Forced tick to provide QS. */ #ifdef CONFIG_RCU_FAST_NO_HZ -bool all_lazy;/* All CPU's CBs lazy at idle start? */ unsigned long last_accelerate;/* Last jiffy CBs were accelerated. */ unsigned long last_advance_all;/* Last jiffy CBs were all advanced. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */
kernel/rcu/tree_plugin.h +9 -33
80 lines not on this page · raw message
kernel/rcu/tree_stall.h +2 -4
@@ -263,11 +263,9 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) {struct rcu_data *rdp = &per_cpu (rcu_data, cpu); -sprintf (cp,"last_accelerate: %04lx/%04lx, Nonlazy posted: %c%c%c" , +sprintf (cp,"last_accelerate: %04lx/%04lx dyntick_enabled: %d" , rdp->last_accelerate &0xffff , jiffies &0xffff , -".l" [rdp->all_lazy], -".L" [!rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)], -".D" [!!rdp->tick_nohz_enabled_snap]); + !!rdp->tick_nohz_enabled_snap); }#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerRe: [PATCH 4/5] rcu: Remove kfree_rcu() special casing and lazy handling
Quoted 5 lines · Joel Fernandes (Google)
Remove
kfree_rcu()special casing and lazy handling from RCU. For Tiny RCU we fold the special handling into just Tiny RCU code.Suggested-by Paul E. McKenney <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>
Nice! Several comments inline below.
Thanx, Paul
Quoted 41 lines
--- Documentation/RCU/stallwarn.txt | 13 +++------ include/linux/rcu_segcblist.h | 2 -- include/trace/events/rcu.h | 32 +++++++++------------- kernel/rcu/rcu.h | 27 ------------------- kernel/rcu/rcu_segcblist.c | 25 +++-------------- kernel/rcu/rcu_segcblist.h | 25 ++--------------- kernel/rcu/srcutree.c | 4 +-- kernel/rcu/tiny.c | 29 +++++++++++++++++++- kernel/rcu/tree.c | 48 ++++++++++++++++++++++++--------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 42 +++++++---------------------- kernel/rcu/tree_stall.h | 6 ++--- 12 files changed, 97 insertions(+), 157 deletions(-)Documentation/RCU/stallwarn.txt +2
-8@@ -225,18 +225,12 @@ an estimate of the total number of RCU callbacks queued across all CPUs In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed for each CPU: - 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D + 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1 The "last_accelerate:" prints the low-order 16 bits (in hex) of the jiffies counter when this CPU last invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from -rcu_prepare_for_idle(). The "Nonlazy posted:" indicates lazy-callback -status, so that an "l" indicates that all callbacks were lazy at the start -of the last idle period and an "L" indicates that there are currently -no non-lazy callbacks (in both cases, "." is printed otherwise, as -shown above) and "D" indicates that dyntick-idle processing is enabled -("." is printed otherwise, for example, if disabled via the "nohz=" -kernel boot parameter). +rcu_prepare_for_idle(). If the grace period ends just as the stall warning starts printing, there will be a spurious stall-warning message, which will include
Documentation/RCU/stallwarn.txt possible for a zero-jiffy stall to be flagged in this case, depending · line 247
Quoted 6 lines
Documentation/RCU/stallwarn.txt +2
-1@@ -249,7 +243,8 @@ possible for a zero-jiffy stall to be flagged in this case, depending on how the stall warning and the grace-period initialization happen to interact. Please note that it is not possible to entirely eliminate this sort of false positive without resorting to things like stop_machine(), -which is overkill for this sort of problem. +which is overkill for this sort of problem. "dyntick_enabled: 1" indicates that +dyntick-idle processing is enabled.
OK, I'll bite... Why not add the sentence to the earlier paragraph that used to discuss its "D" counterpart?
Quoted 151 lines
Documentation/RCU/stallwarn.txt +0
-0If all CPUs and tasks have passed through quiescent states, but the grace period has nevertheless failed to end, the stall-warning splatinclude/linux/rcu_segcblist.h +0
-2@@ -22,7 +22,6 @@ struct rcu_cblist {struct rcu_head *head;struct rcu_head **tail;long len; -long len_lazy; };#define RCU_CBLIST_INITIALIZER (n) { .head = NULL, .tail = &n.head }@@ -73,7 +72,6 @@ struct rcu_segcblist {#else long len;#endif -long len_lazy;u8 enabled;u8 offloaded; };include/trace/events/rcu.h +12
-20@@ -474,16 +474,14 @@ TRACE_EVENT_RCU(rcu_dyntick,*/ TRACE_EVENT_RCU(rcu_callback, - TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, - long qlen), + TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen), - TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(void *, func) - __field(long, qlen_lazy) __field(long, qlen) ),@@ -491,13 +489,12 @@ TRACE_EVENT_RCU(rcu_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->func = rhp->func; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ps %ld/%ld" , +TP_printk ("%s rhp=%p func=%ps %ld" , __entry->rcuname, __entry->rhp, __entry->func, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/* @@ -511,15 +508,14 @@ TRACE_EVENT_RCU(rcu_callback, TRACE_EVENT_RCU(rcu_kfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, - long qlen_lazy, long qlen), + long qlen), - TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, offset, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(unsigned long, offset) - __field(long, qlen_lazy) __field(long, qlen) ),@@ -527,13 +523,12 @@ TRACE_EVENT_RCU(rcu_kfree_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->offset = offset; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ld %ld/%ld" , +TP_printk ("%s rhp=%p func=%ld %ld" , __entry->rcuname, __entry->rhp, __entry->offset, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/* @@ -545,27 +540,24 @@ TRACE_EVENT_RCU(rcu_kfree_callback,*/ TRACE_EVENT_RCU (rcu_batch_start, - TP_PROTO(const char *rcuname, long qlen_lazy, long qlen, long blimit), + TP_PROTO(const char *rcuname, long qlen, long blimit), -TP_ARGS (rcuname, qlen_lazy, qlen, blimit), +TP_ARGS (rcuname, qlen, blimit),TP_STRUCT__entry ( __field(const char *, rcuname) - __field(long, qlen_lazy) __field(long, qlen)__field (long, blimit) ),TP_fast_assign ( __entry->rcuname = rcuname; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; __entry->blimit = blimit; ), -TP_printk ("%s CBs=%ld/%ld bl=%ld" , - __entry->rcuname, __entry->qlen_lazy, __entry->qlen, - __entry->blimit) +TP_printk ("%s CBs=%ld bl=%ld" , + __entry->rcuname, __entry->qlen, __entry->blimit) );/* kernel/rcu/rcu.h +0
-16@@ -198,33 +198,6 @@ static inline void debug_rcu_head_unqueue(struct rcu_head *head) }#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */ -void kfree (const void *); - -/* -* Reclaim the specified callback, either by invoking it (non-lazy case) -* or freeing it directly (lazy case). Return true if lazy, false otherwise. -*/ -static inline bool __rcu_reclaim (const char *rn,struct rcu_head *head) -{ -rcu_callback_t f; -unsigned long offset = (unsigned long )head->func; - -rcu_lock_acquire (&rcu_callback_map); -if (__is_kfree_rcu_offset (offset)) { -trace_rcu_invoke_kfree_callback (rn, head, offset); -kfree ((void *)head - offset); -rcu_lock_release (&rcu_callback_map);
kernel/rcu/rcu.h static inline void debug_rcu_head_unqueue(struct rcu_head *head) · line 226
Quoted 10 lines
kernel/rcu/rcu.h +0
-10-return true; - }else { -trace_rcu_invoke_callback (rn, head); - f = head->func; -WRITE_ONCE (head->func, (rcu_callback_t )0L ); -f (head); -rcu_lock_release (&rcu_callback_map); -return false; - } -}
Nice to see this one go! ;-)
Quoted 210 lines
kernel/rcu/rcu.h +0
-0#ifdef CONFIG_RCU_STALL_COMMON extern int rcu_cpu_stall_ftrace_dump;kernel/rcu/rcu_segcblist.c +3
-22@@ -20,14 +20,10 @@ void rcu_cblist_init(struct rcu_cblist *rclp) rclp->head =NULL ; rclp->tail = &rclp->head; rclp->len =0 ; - rclp->len_lazy =0 ; }/* * Enqueue an rcu_head structure onto the specified callback list. -* This function assumes that the callback is non-lazy because it -* is intended for use by no-CBs CPUs, which do not distinguish -* between lazy and non-lazy RCU callbacks. */ void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp) {@@ -54,7 +50,6 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,else drclp->tail = &drclp->head; drclp->len = srclp->len; - drclp->len_lazy = srclp->len_lazy;if (!rhp) {rcu_cblist_init (srclp); }else {@@ -62,16 +57,12 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, srclp->head = rhp; srclp->tail = &rhp->next;WRITE_ONCE (srclp->len,1 ); - srclp->len_lazy =0 ; } }/* * Dequeue the oldest rcu_head structure from the specified callback -* list. This function assumes that the callback is non-lazy, but -* the caller can later invoke rcu_cblist_dequeued_lazy() if it -* finds otherwise (and if it cares about laziness). This allows -* different users to have different ways of determining laziness. +* list. */ struct rcu_head *rcu_cblist_dequeue (struct rcu_cblist *rclp) {@@ -161,7 +152,6 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)for (i =0 ; i <RCU_CBLIST_NSEGS ; i++) rsclp->tails[i] = &rsclp->head;rcu_segcblist_set_len (rsclp,0 ); - rsclp->len_lazy =0 ; rsclp->enabled =1 ; }@@ -173,7 +163,6 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) {WARN_ON_ONCE (!rcu_segcblist_empty (rsclp));WARN_ON_ONCE (rcu_segcblist_n_cbs (rsclp)); -WARN_ON_ONCE (rcu_segcblist_n_lazy_cbs (rsclp)); rsclp->enabled =0 ; }@@ -253,11 +242,9 @@ bool rcu_segcblist_nextgp(struct rcu_segcblist *rsclp, unsigned long *lp)* absolutely not OK for it to ever miss posting a callback. */ void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is enqueued. */ rhp->next =NULL ;WRITE_ONCE (*rsclp->tails[RCU_NEXT_TAIL ], rhp);@@ -275,15 +262,13 @@ void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,* period. You have been warned. */ bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {int i;if (rcu_segcblist_n_cbs (rsclp) ==0 )return false;rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is entrained. */ rhp->next =NULL ;for (i =RCU_NEXT_TAIL ; i >RCU_DONE_TAIL ; i--)@@ -307,8 +292,6 @@ bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rclp->len_lazy += rsclp->len_lazy; - rsclp->len_lazy =0 ; rclp->len =rcu_segcblist_xchg_len (rsclp,0 ); }@@ -361,9 +344,7 @@ void rcu_segcblist_extract_pend_cbs(struct rcu_segcblist *rsclp,void rcu_segcblist_insert_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rsclp->len_lazy += rclp->len_lazy;rcu_segcblist_add_len (rsclp, rclp->len); - rclp->len_lazy =0 ; rclp->len =0 ; }kernel/rcu/rcu_segcblist.h +2
-23@@ -15,15 +15,6 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp)return READ_ONCE (rclp->len); } -/* -* Account for the fact that a previously dequeued callback turned out -* to be marked as lazy. -*/ -static inline void rcu_cblist_dequeued_lazy (struct rcu_cblist *rclp) -{ - rclp->len_lazy--; -} -void rcu_cblist_init (struct rcu_cblist *rclp);void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp); void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,@@ -59,18 +50,6 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)#endif } -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_lazy_cbs (struct rcu_segcblist *rsclp) -{ -return rsclp->len_lazy; -} - -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_nonlazy_cbs (struct rcu_segcblist *rsclp) -{ -return rcu_segcblist_n_cbs (rsclp) - rsclp->len_lazy; -} -/* * Is the specified rcu_segcblist enabled, for example, not corresponding * to an offline CPU? @@ -106,9 +85,9 @@ struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_pend_cb (struct rcu_segcblist *rsclp);bool rcu_segcblist_nextgp (struct rcu_segcblist *rsclp,unsigned long *lp);void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp); void rcu_segcblist_extract_done_cbs(struct rcu_segcblist *rsclp,kernel/rcu/srcutree.c +2
-2@@ -853,7 +853,7 @@ static void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,local_irq_save (flags); sdp =this_cpu_ptr (ssp->sda);spin_lock_rcu_node (sdp); -rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp, false); +rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp);rcu_segcblist_advance (&sdp->srcu_cblist,rcu_seq_current (&ssp->srcu_gp_seq)); s =rcu_seq_snap (&ssp->srcu_gp_seq);@@ -1052,7 +1052,7 @@ void srcu_barrier(struct srcu_struct *ssp) sdp->srcu_barrier_head.func = srcu_barrier_cb;debug_rcu_head_queue (&sdp->srcu_barrier_head);if (!rcu_segcblist_entrain (&sdp->srcu_cblist, - &sdp->srcu_barrier_head,0 )) { + &sdp->srcu_barrier_head)) {debug_rcu_head_unqueue (&sdp->srcu_barrier_head);atomic_dec (&ssp->srcu_barrier_cpu_cnt); }kernel/rcu/tiny.c +16
-0@@ -73,6 +73,33 @@ void rcu_sched_clock_irq(int user) } } +void kfree (const void *); + +/* +* Reclaim the specified callback, either by invoking it for non-kfree cases or +* freeing it directly (for kfree). Return true if kfreeing, false otherwise. +*/ +static inline bool rcu_reclaim_tiny (struct rcu_head *head) +{ +rcu_callback_t f; +unsigned long offset = (unsigned long )head->func; + +rcu_lock_acquire (&rcu_callback_map); +if (__is_kfree_rcu_offset (offset)) { +trace_rcu_invoke_kfree_callback ("" , head, offset); +kfree ((void *)head - offset); +rcu_lock_release (&rcu_callback_map);
kernel/rcu/tiny.c void rcu_sched_clock_irq(int user) · line 101
Quoted 10 lines
kernel/rcu/tiny.c +10
-0+return true; + } + +trace_rcu_invoke_callback ("" , head); + f = head->func; +WRITE_ONCE (head->func, (rcu_callback_t )0L ); +f (head); +rcu_lock_release (&rcu_callback_map); +return false; +}
Even if it did show up here in the meantime. ;-)
Quoted 26 lines
kernel/rcu/tiny.c +1
-1/* Invoke the RCU callbacks whose grace period has elapsed. */ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused) {@@ -100,7 +127,7 @@ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unusedprefetch (next);debug_rcu_head_unqueue (list);local_bh_disable (); -__rcu_reclaim ("" , list); +rcu_reclaim_tiny (list);local_bh_enable (); list = next; }kernel/rcu/tree.c +6
-0@@ -2129,6 +2129,23 @@ int rcutree_dead_cpu(unsigned int cpu)return 0 ; } +/* +* Reclaim the specified callback. +*/ +static inline void rcu_reclaim_tree (const char *rn,struct rcu_head *head) +{ +rcu_callback_t f;
kernel/rcu/tree.c int rcutree_dead_cpu(unsigned int cpu) · line 2147
Quoted 10 lines
kernel/rcu/tree.c +10
-0+ +rcu_lock_acquire (&rcu_callback_map); +trace_rcu_invoke_callback (rn, head); + + f = head->func; +WRITE_ONCE (head->func, (rcu_callback_t )0L ); +f (head); + +rcu_lock_release (&rcu_callback_map); +}
Might be worth inlining this one, as you did for kfree_rcu().
kernel/rcu/tree_plugin.h static void rcu_prepare_for_idle(void)
Quoted 125 lines
kernel/rcu/tree.c +18
-13/* * Invoke any RCU callbacks that have made it to the end of their grace * period. Thottle as specified by rdp->blimit. @@ -2146,7 +2163,6 @@ static void rcu_do_batch(struct rcu_data *rdp)/* If no callbacks are ready, just return. */ if (!rcu_segcblist_ready_cbs (&rdp->cblist)) {trace_rcu_batch_start (rcu_state.name, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist),0 ); trace_rcu_batch_end(rcu_state.name, 0, !rcu_segcblist_empty(&rdp->cblist),@@ -2168,7 +2184,6 @@ static void rcu_do_batch(struct rcu_data *rdp)if (unlikely (bl >100 )) tlimit =local_clock () + rcu_resched_ns;trace_rcu_batch_start (rcu_state.name, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist), bl);rcu_segcblist_extract_done_cbs (&rdp->cblist, &rcl);if (offloaded)@@ -2181,8 +2196,8 @@ static void rcu_do_batch(struct rcu_data *rdp) rhp =rcu_cblist_dequeue (&rcl);for (; rhp; rhp =rcu_cblist_dequeue (&rcl)) {debug_rcu_head_unqueue (rhp); -if (__rcu_reclaim (rcu_state.name, rhp)) -rcu_cblist_dequeued_lazy (&rcl); +rcu_reclaim_tree (rcu_state.name, rhp); +/* * Stop only if limit reached and CPU has something to do. * Note: The rcl structure counts down from zero. @@ -2585,7 +2600,7 @@ static void rcu_leak_callback(struct rcu_head *rhp)* is expected to specify a CPU. */ static void -__call_rcu (struct rcu_head *head,rcu_callback_t func,bool lazy) +__call_rcu (struct rcu_head *head,rcu_callback_t func) {unsigned long flags;struct rcu_data *rdp;@@ -2620,18 +2635,17 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)if (rcu_segcblist_empty (&rdp->cblist))rcu_segcblist_init (&rdp->cblist); } +if (rcu_nocb_try_bypass (rdp, head, &was_alldone, flags))return ;// Enqueued onto ->nocb_bypass, so just leave. /* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */ -rcu_segcblist_enqueue (&rdp->cblist, head, lazy); +rcu_segcblist_enqueue (&rdp->cblist, head);if (__is_kfree_rcu_offset ((unsigned long )func))trace_rcu_kfree_callback (rcu_state.name, head, (unsigned long )func, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist));else trace_rcu_callback (rcu_state.name, head, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist));/* Go handle any RCU core processing required. */ @@ -2681,7 +2695,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)*/ void call_rcu (struct rcu_head *head,rcu_callback_t func) { -__call_rcu (head, func,0 ); +__call_rcu (head, func); }EXPORT_SYMBOL_GPL (call_rcu);@@ -2755,10 +2769,18 @@ static void kfree_rcu_work(struct work_struct *work)* access is Ok. */ for (; head; head = next) { +unsigned long offset = (unsigned long )head->func; + next = head->next; -/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head); -__rcu_reclaim (rcu_state.name, head); +rcu_lock_acquire (&rcu_callback_map); +trace_rcu_invoke_kfree_callback (rcu_state.name, head, offset); + +/* Could be possible to optimize with kfree_bulk in future */ +kfree ((void *)head - offset); + +rcu_lock_release (&rcu_callback_map);cond_resched_tasks_rcu_qs (); } }@@ -2840,7 +2862,7 @@ static void kfree_rcu_monitor(struct work_struct *work)*/ void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) { -__call_rcu (head, func,1 ); +__call_rcu (head, func); }EXPORT_SYMBOL_GPL (kfree_call_rcu_nobatch);@@ -3094,7 +3116,7 @@ static void rcu_barrier_func(void *unused)debug_rcu_head_queue (&rdp->barrier_head);rcu_nocb_lock (rdp);WARN_ON_ONCE (!rcu_nocb_flush_bypass (rdp,NULL , jiffies)); -if (rcu_segcblist_entrain (&rdp->cblist, &rdp->barrier_head,0 )) { +if (rcu_segcblist_entrain (&rdp->cblist, &rdp->barrier_head)) {atomic_inc (&rcu_state.barrier_cpu_count); }else {debug_rcu_head_unqueue (&rdp->barrier_head);kernel/rcu/tree.h +0
-1@@ -183,7 +183,6 @@ struct rcu_data {bool rcu_urgent_qs;/* GP old need light quiescent state. */ bool rcu_forced_tick;/* Forced tick to provide QS. */ #ifdef CONFIG_RCU_FAST_NO_HZ -bool all_lazy;/* All CPU's CBs lazy at idle start? */ unsigned long last_accelerate;/* Last jiffy CBs were accelerated. */ unsigned long last_advance_all;/* Last jiffy CBs were all advanced. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ kernel/rcu/tree_plugin.h +0
-0@@ -1274,21 +1274,15 @@ static void rcu_prepare_for_idle(void)
We now have only one preprocessor symbol (as opposed to the erroneous "three proprocessor symbols" in my original).
kernel/rcu/tree_plugin.h static void rcu_prepare_for_idle(void) · line 1278
Quoted 8 lines
kernel/rcu/tree_plugin.h +0
-3* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your * system. And if you are -that- concerned about energy efficiency, * just power the system down and be done with it! -* RCU_IDLE_LAZY_GP_DELAY gives the number of jiffies that a CPU is -* permitted to sleep in dyntick-idle mode with only lazy RCU -* callbacks pending. Setting this too high can OOM your system. * * The values below work well in practice. If future workloads require
"value below works"
Quoted 28 lines
kernel/rcu/tree_plugin.h +1
-6* adjustment, they can be converted into kernel config parameters, though * making the state machine smarter might be a better option. */ #define RCU_IDLE_GP_DELAY 4/* Roughly one grace period. */ -#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ)/* Roughly six seconds. */ static int rcu_idle_gp_delay =RCU_IDLE_GP_DELAY ;module_param (rcu_idle_gp_delay, int,0644 ); -static int rcu_idle_lazy_gp_delay =RCU_IDLE_LAZY_GP_DELAY ; -module_param (rcu_idle_lazy_gp_delay, int,0644 );/* * Try to advance callbacks on the current CPU, but only if it has been @@ -1327,8 +1321,7 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void)/* * Allow the CPU to enter dyntick-idle mode unless it has callbacks ready * to invoke. If the CPU has callbacks, try to advance them. Tell the -* caller to set the timeout based on whether or not there are non-lazy -* callbacks. +* caller about what to set the timeout. * * The caller must have disabled interrupts. */ @@ -1354,25 +1347,19 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt) } rdp->last_accelerate = jiffies; -/* Request timer delay depending on laziness, and round. */
kernel/rcu/tree_plugin.h int rcu_needs_cpu(u64 basemono, u64 *nextevt) · line 1352
Quoted 10 lines
kernel/rcu/tree_plugin.h +3
-7- rdp->all_lazy = !rcu_segcblist_n_nonlazy_cbs (&rdp->cblist); -if (rdp->all_lazy) { - dj =round_jiffies (rcu_idle_lazy_gp_delay + jiffies) - jiffies; - }else { - dj =round_up (rcu_idle_gp_delay + jiffies, - rcu_idle_gp_delay) - jiffies; - } +/* Request timer and round. */ + dj =round_up (rcu_idle_gp_delay + jiffies, + rcu_idle_gp_delay) - jiffies;
I bet that this now fits onto one line.
Quoted 58 lines
kernel/rcu/tree_plugin.h +5
-17+ *nextevt = basemono + dj *TICK_NSEC ;return 0 ; }/* -* Prepare a CPU for idle from an RCU perspective. The first major task -* is to sense whether nohz mode has been enabled or disabled via sysfs. -* The second major task is to check to see if a non-lazy callback has -* arrived at a CPU that previously had only lazy callbacks. The third -* major task is to accelerate (that is, assign grace-period numbers to) -* any recently arrived callbacks. +* Prepare a CPU for idle from an RCU perspective. The first major task is to +* sense whether nohz mode has been enabled or disabled via sysfs. The second +* major task is to accelerate (that is, assign grace-period numbers to) any +* recently arrived callbacks. * * The caller must have disabled interrupts. */ @@ -1398,17 +1385,6 @@ static void rcu_prepare_for_idle(void)if (!tne)return ; -/* -* If a non-lazy callback arrived at a CPU having only lazy -* callbacks, invoke RCU core for the side-effect of recalculating -* idle duration on re-entry to idle. -*/ -if (rdp->all_lazy &&rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)) { - rdp->all_lazy = false; -invoke_rcu_core (); -return ; - } -/* * If we have not yet accelerated this jiffy, accelerate all * callbacks on this CPU. kernel/rcu/tree_stall.h +2
-4@@ -263,11 +263,9 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) {struct rcu_data *rdp = &per_cpu (rcu_data, cpu); -sprintf (cp,"last_accelerate: %04lx/%04lx, Nonlazy posted: %c%c%c" , +sprintf (cp,"last_accelerate: %04lx/%04lx dyntick_enabled: %d" , rdp->last_accelerate &0xffff , jiffies &0xffff , -".l" [rdp->all_lazy], -".L" [!rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)], -".D" [!!rdp->tick_nohz_enabled_snap]); + !!rdp->tick_nohz_enabled_snap); }#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */ Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH 4/5] rcu: Remove kfree_rcu() special casing and lazy handling
All comments addressed on this one, thanks!
- Joel
Quoted 727 lines · Paul E. McKenney
Quoted 5 lines · Joel Fernandes (Google)
Remove
kfree_rcu()special casing and lazy handling from RCU. For Tiny RCU we fold the special handling into just Tiny RCU code.Suggested-by Paul E. McKenney <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>Nice! Several comments inline below.
Thanx, Paul
Quoted 41 lines
--- Documentation/RCU/stallwarn.txt | 13 +++------ include/linux/rcu_segcblist.h | 2 -- include/trace/events/rcu.h | 32 +++++++++------------- kernel/rcu/rcu.h | 27 ------------------- kernel/rcu/rcu_segcblist.c | 25 +++-------------- kernel/rcu/rcu_segcblist.h | 25 ++--------------- kernel/rcu/srcutree.c | 4 +-- kernel/rcu/tiny.c | 29 +++++++++++++++++++- kernel/rcu/tree.c | 48 ++++++++++++++++++++++++--------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 42 +++++++---------------------- kernel/rcu/tree_stall.h | 6 ++--- 12 files changed, 97 insertions(+), 157 deletions(-)Documentation/RCU/stallwarn.txt +2
-8@@ -225,18 +225,12 @@ an estimate of the total number of RCU callbacks queued across all CPUs In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed for each CPU: - 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D + 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1 The "last_accelerate:" prints the low-order 16 bits (in hex) of the jiffies counter when this CPU last invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from -rcu_prepare_for_idle(). The "Nonlazy posted:" indicates lazy-callback -status, so that an "l" indicates that all callbacks were lazy at the start -of the last idle period and an "L" indicates that there are currently -no non-lazy callbacks (in both cases, "." is printed otherwise, as -shown above) and "D" indicates that dyntick-idle processing is enabled -("." is printed otherwise, for example, if disabled via the "nohz=" -kernel boot parameter). +rcu_prepare_for_idle(). If the grace period ends just as the stall warning starts printing, there will be a spurious stall-warning message, which will includeDocumentation/RCU/stallwarn.txt possible for a zero-jiffy stall to be flagged in this case, depending · line 247
Quoted 6 lines
Documentation/RCU/stallwarn.txt +2
-1@@ -249,7 +243,8 @@ possible for a zero-jiffy stall to be flagged in this case, depending on how the stall warning and the grace-period initialization happen to interact. Please note that it is not possible to entirely eliminate this sort of false positive without resorting to things like stop_machine(), -which is overkill for this sort of problem. +which is overkill for this sort of problem. "dyntick_enabled: 1" indicates that +dyntick-idle processing is enabled.OK, I'll bite... Why not add the sentence to the earlier paragraph that used to discuss its "D" counterpart?
Quoted 151 lines
Documentation/RCU/stallwarn.txt +0
-0If all CPUs and tasks have passed through quiescent states, but the grace period has nevertheless failed to end, the stall-warning splatinclude/linux/rcu_segcblist.h +0
-2@@ -22,7 +22,6 @@ struct rcu_cblist {struct rcu_head *head;struct rcu_head **tail;long len; -long len_lazy; };#define RCU_CBLIST_INITIALIZER (n) { .head = NULL, .tail = &n.head }@@ -73,7 +72,6 @@ struct rcu_segcblist {#else long len;#endif -long len_lazy;u8 enabled;u8 offloaded; };include/trace/events/rcu.h +12
-20@@ -474,16 +474,14 @@ TRACE_EVENT_RCU(rcu_dyntick,*/ TRACE_EVENT_RCU(rcu_callback, - TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, - long qlen), + TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen), - TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(void *, func) - __field(long, qlen_lazy) __field(long, qlen) ),@@ -491,13 +489,12 @@ TRACE_EVENT_RCU(rcu_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->func = rhp->func; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ps %ld/%ld" , +TP_printk ("%s rhp=%p func=%ps %ld" , __entry->rcuname, __entry->rhp, __entry->func, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/* @@ -511,15 +508,14 @@ TRACE_EVENT_RCU(rcu_callback, TRACE_EVENT_RCU(rcu_kfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, - long qlen_lazy, long qlen), + long qlen), - TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen), + TP_ARGS(rcuname, rhp, offset, qlen), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(unsigned long, offset) - __field(long, qlen_lazy) __field(long, qlen) ),@@ -527,13 +523,12 @@ TRACE_EVENT_RCU(rcu_kfree_callback, __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->offset = offset; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; ), -TP_printk ("%s rhp=%p func=%ld %ld/%ld" , +TP_printk ("%s rhp=%p func=%ld %ld" , __entry->rcuname, __entry->rhp, __entry->offset, - __entry->qlen_lazy, __entry->qlen) + __entry->qlen) );/* @@ -545,27 +540,24 @@ TRACE_EVENT_RCU(rcu_kfree_callback,*/ TRACE_EVENT_RCU (rcu_batch_start, - TP_PROTO(const char *rcuname, long qlen_lazy, long qlen, long blimit), + TP_PROTO(const char *rcuname, long qlen, long blimit), -TP_ARGS (rcuname, qlen_lazy, qlen, blimit), +TP_ARGS (rcuname, qlen, blimit),TP_STRUCT__entry ( __field(const char *, rcuname) - __field(long, qlen_lazy) __field(long, qlen)__field (long, blimit) ),TP_fast_assign ( __entry->rcuname = rcuname; - __entry->qlen_lazy = qlen_lazy; __entry->qlen = qlen; __entry->blimit = blimit; ), -TP_printk ("%s CBs=%ld/%ld bl=%ld" , - __entry->rcuname, __entry->qlen_lazy, __entry->qlen, - __entry->blimit) +TP_printk ("%s CBs=%ld bl=%ld" , + __entry->rcuname, __entry->qlen, __entry->blimit) );/* kernel/rcu/rcu.h +0
-16@@ -198,33 +198,6 @@ static inline void debug_rcu_head_unqueue(struct rcu_head *head) }#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */ -void kfree (const void *); - -/* -* Reclaim the specified callback, either by invoking it (non-lazy case) -* or freeing it directly (lazy case). Return true if lazy, false otherwise. -*/ -static inline bool __rcu_reclaim (const char *rn,struct rcu_head *head) -{ -rcu_callback_t f; -unsigned long offset = (unsigned long )head->func; - -rcu_lock_acquire (&rcu_callback_map); -if (__is_kfree_rcu_offset (offset)) { -trace_rcu_invoke_kfree_callback (rn, head, offset); -kfree ((void *)head - offset); -rcu_lock_release (&rcu_callback_map);kernel/rcu/rcu.h static inline void debug_rcu_head_unqueue(struct rcu_head *head) · line 226
Quoted 10 lines
kernel/rcu/rcu.h +0
-10-return true; - }else { -trace_rcu_invoke_callback (rn, head); - f = head->func; -WRITE_ONCE (head->func, (rcu_callback_t )0L ); -f (head); -rcu_lock_release (&rcu_callback_map); -return false; - } -}Nice to see this one go! ;-)
Quoted 210 lines
kernel/rcu/rcu.h +0
-0#ifdef CONFIG_RCU_STALL_COMMON extern int rcu_cpu_stall_ftrace_dump;kernel/rcu/rcu_segcblist.c +3
-22@@ -20,14 +20,10 @@ void rcu_cblist_init(struct rcu_cblist *rclp) rclp->head =NULL ; rclp->tail = &rclp->head; rclp->len =0 ; - rclp->len_lazy =0 ; }/* * Enqueue an rcu_head structure onto the specified callback list. -* This function assumes that the callback is non-lazy because it -* is intended for use by no-CBs CPUs, which do not distinguish -* between lazy and non-lazy RCU callbacks. */ void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp) {@@ -54,7 +50,6 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,else drclp->tail = &drclp->head; drclp->len = srclp->len; - drclp->len_lazy = srclp->len_lazy;if (!rhp) {rcu_cblist_init (srclp); }else {@@ -62,16 +57,12 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, srclp->head = rhp; srclp->tail = &rhp->next;WRITE_ONCE (srclp->len,1 ); - srclp->len_lazy =0 ; } }/* * Dequeue the oldest rcu_head structure from the specified callback -* list. This function assumes that the callback is non-lazy, but -* the caller can later invoke rcu_cblist_dequeued_lazy() if it -* finds otherwise (and if it cares about laziness). This allows -* different users to have different ways of determining laziness. +* list. */ struct rcu_head *rcu_cblist_dequeue (struct rcu_cblist *rclp) {@@ -161,7 +152,6 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)for (i =0 ; i <RCU_CBLIST_NSEGS ; i++) rsclp->tails[i] = &rsclp->head;rcu_segcblist_set_len (rsclp,0 ); - rsclp->len_lazy =0 ; rsclp->enabled =1 ; }@@ -173,7 +163,6 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) {WARN_ON_ONCE (!rcu_segcblist_empty (rsclp));WARN_ON_ONCE (rcu_segcblist_n_cbs (rsclp)); -WARN_ON_ONCE (rcu_segcblist_n_lazy_cbs (rsclp)); rsclp->enabled =0 ; }@@ -253,11 +242,9 @@ bool rcu_segcblist_nextgp(struct rcu_segcblist *rsclp, unsigned long *lp)* absolutely not OK for it to ever miss posting a callback. */ void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is enqueued. */ rhp->next =NULL ;WRITE_ONCE (*rsclp->tails[RCU_NEXT_TAIL ], rhp);@@ -275,15 +262,13 @@ void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,* period. You have been warned. */ bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy) +struct rcu_head *rhp) {int i;if (rcu_segcblist_n_cbs (rsclp) ==0 )return false;rcu_segcblist_inc_len (rsclp); -if (lazy) - rsclp->len_lazy++;smp_mb ();/* Ensure counts are updated before callback is entrained. */ rhp->next =NULL ;for (i =RCU_NEXT_TAIL ; i >RCU_DONE_TAIL ; i--)@@ -307,8 +292,6 @@ bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rclp->len_lazy += rsclp->len_lazy; - rsclp->len_lazy =0 ; rclp->len =rcu_segcblist_xchg_len (rsclp,0 ); }@@ -361,9 +344,7 @@ void rcu_segcblist_extract_pend_cbs(struct rcu_segcblist *rsclp,void rcu_segcblist_insert_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp) { - rsclp->len_lazy += rclp->len_lazy;rcu_segcblist_add_len (rsclp, rclp->len); - rclp->len_lazy =0 ; rclp->len =0 ; }kernel/rcu/rcu_segcblist.h +2
-23@@ -15,15 +15,6 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp)return READ_ONCE (rclp->len); } -/* -* Account for the fact that a previously dequeued callback turned out -* to be marked as lazy. -*/ -static inline void rcu_cblist_dequeued_lazy (struct rcu_cblist *rclp) -{ - rclp->len_lazy--; -} -void rcu_cblist_init (struct rcu_cblist *rclp);void rcu_cblist_enqueue (struct rcu_cblist *rclp,struct rcu_head *rhp); void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,@@ -59,18 +50,6 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)#endif } -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_lazy_cbs (struct rcu_segcblist *rsclp) -{ -return rsclp->len_lazy; -} - -/* Return number of lazy callbacks in segmented callback list. */ -static inline long rcu_segcblist_n_nonlazy_cbs (struct rcu_segcblist *rsclp) -{ -return rcu_segcblist_n_cbs (rsclp) - rsclp->len_lazy; -} -/* * Is the specified rcu_segcblist enabled, for example, not corresponding * to an offline CPU? @@ -106,9 +85,9 @@ struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_pend_cb (struct rcu_segcblist *rsclp);bool rcu_segcblist_nextgp (struct rcu_segcblist *rsclp,unsigned long *lp);void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);bool rcu_segcblist_entrain (struct rcu_segcblist *rsclp, -struct rcu_head *rhp,bool lazy); +struct rcu_head *rhp);void rcu_segcblist_extract_count (struct rcu_segcblist *rsclp,struct rcu_cblist *rclp); void rcu_segcblist_extract_done_cbs(struct rcu_segcblist *rsclp,kernel/rcu/srcutree.c +2
-2@@ -853,7 +853,7 @@ static void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,local_irq_save (flags); sdp =this_cpu_ptr (ssp->sda);spin_lock_rcu_node (sdp); -rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp, false); +rcu_segcblist_enqueue (&sdp->srcu_cblist, rhp);rcu_segcblist_advance (&sdp->srcu_cblist,rcu_seq_current (&ssp->srcu_gp_seq)); s =rcu_seq_snap (&ssp->srcu_gp_seq);@@ -1052,7 +1052,7 @@ void srcu_barrier(struct srcu_struct *ssp) sdp->srcu_barrier_head.func = srcu_barrier_cb;debug_rcu_head_queue (&sdp->srcu_barrier_head);if (!rcu_segcblist_entrain (&sdp->srcu_cblist, - &sdp->srcu_barrier_head,0 )) { + &sdp->srcu_barrier_head)) {debug_rcu_head_unqueue (&sdp->srcu_barrier_head);atomic_dec (&ssp->srcu_barrier_cpu_cnt); }kernel/rcu/tiny.c +16
-0@@ -73,6 +73,33 @@ void rcu_sched_clock_irq(int user) } } +void kfree (const void *); + +/* +* Reclaim the specified callback, either by invoking it for non-kfree cases or +* freeing it directly (for kfree). Return true if kfreeing, false otherwise. +*/ +static inline bool rcu_reclaim_tiny (struct rcu_head *head) +{ +rcu_callback_t f; +unsigned long offset = (unsigned long )head->func; + +rcu_lock_acquire (&rcu_callback_map); +if (__is_kfree_rcu_offset (offset)) { +trace_rcu_invoke_kfree_callback ("" , head, offset); +kfree ((void *)head - offset); +rcu_lock_release (&rcu_callback_map);kernel/rcu/tiny.c void rcu_sched_clock_irq(int user) · line 101
Quoted 10 lines
kernel/rcu/tiny.c +10
-0+return true; + } + +trace_rcu_invoke_callback ("" , head); + f = head->func; +WRITE_ONCE (head->func, (rcu_callback_t )0L ); +f (head); +rcu_lock_release (&rcu_callback_map); +return false; +}Even if it did show up here in the meantime. ;-)
Quoted 26 lines
kernel/rcu/tiny.c +1
-1/* Invoke the RCU callbacks whose grace period has elapsed. */ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused) {@@ -100,7 +127,7 @@ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unusedprefetch (next);debug_rcu_head_unqueue (list);local_bh_disable (); -__rcu_reclaim ("" , list); +rcu_reclaim_tiny (list);local_bh_enable (); list = next; }kernel/rcu/tree.c +6
-0@@ -2129,6 +2129,23 @@ int rcutree_dead_cpu(unsigned int cpu)return 0 ; } +/* +* Reclaim the specified callback. +*/ +static inline void rcu_reclaim_tree (const char *rn,struct rcu_head *head) +{ +rcu_callback_t f;kernel/rcu/tree.c int rcutree_dead_cpu(unsigned int cpu) · line 2147
Quoted 10 lines
kernel/rcu/tree.c +10
-0+ +rcu_lock_acquire (&rcu_callback_map); +trace_rcu_invoke_callback (rn, head); + + f = head->func; +WRITE_ONCE (head->func, (rcu_callback_t )0L ); +f (head); + +rcu_lock_release (&rcu_callback_map); +}Might be worth inlining this one, as you did for
kfree_rcu().kernel/rcu/tree_plugin.h static void rcu_prepare_for_idle(void)
Quoted 125 lines
kernel/rcu/tree.c +18
-13/* * Invoke any RCU callbacks that have made it to the end of their grace * period. Thottle as specified by rdp->blimit. @@ -2146,7 +2163,6 @@ static void rcu_do_batch(struct rcu_data *rdp)/* If no callbacks are ready, just return. */ if (!rcu_segcblist_ready_cbs (&rdp->cblist)) {trace_rcu_batch_start (rcu_state.name, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist),0 ); trace_rcu_batch_end(rcu_state.name, 0, !rcu_segcblist_empty(&rdp->cblist),@@ -2168,7 +2184,6 @@ static void rcu_do_batch(struct rcu_data *rdp)if (unlikely (bl >100 )) tlimit =local_clock () + rcu_resched_ns;trace_rcu_batch_start (rcu_state.name, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist), bl);rcu_segcblist_extract_done_cbs (&rdp->cblist, &rcl);if (offloaded)@@ -2181,8 +2196,8 @@ static void rcu_do_batch(struct rcu_data *rdp) rhp =rcu_cblist_dequeue (&rcl);for (; rhp; rhp =rcu_cblist_dequeue (&rcl)) {debug_rcu_head_unqueue (rhp); -if (__rcu_reclaim (rcu_state.name, rhp)) -rcu_cblist_dequeued_lazy (&rcl); +rcu_reclaim_tree (rcu_state.name, rhp); +/* * Stop only if limit reached and CPU has something to do. * Note: The rcl structure counts down from zero. @@ -2585,7 +2600,7 @@ static void rcu_leak_callback(struct rcu_head *rhp)* is expected to specify a CPU. */ static void -__call_rcu (struct rcu_head *head,rcu_callback_t func,bool lazy) +__call_rcu (struct rcu_head *head,rcu_callback_t func) {unsigned long flags;struct rcu_data *rdp;@@ -2620,18 +2635,17 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)if (rcu_segcblist_empty (&rdp->cblist))rcu_segcblist_init (&rdp->cblist); } +if (rcu_nocb_try_bypass (rdp, head, &was_alldone, flags))return ;// Enqueued onto ->nocb_bypass, so just leave. /* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */ -rcu_segcblist_enqueue (&rdp->cblist, head, lazy); +rcu_segcblist_enqueue (&rdp->cblist, head);if (__is_kfree_rcu_offset ((unsigned long )func))trace_rcu_kfree_callback (rcu_state.name, head, (unsigned long )func, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist));else trace_rcu_callback (rcu_state.name, head, -rcu_segcblist_n_lazy_cbs (&rdp->cblist),rcu_segcblist_n_cbs (&rdp->cblist));/* Go handle any RCU core processing required. */ @@ -2681,7 +2695,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)*/ void call_rcu (struct rcu_head *head,rcu_callback_t func) { -__call_rcu (head, func,0 ); +__call_rcu (head, func); }EXPORT_SYMBOL_GPL (call_rcu);@@ -2755,10 +2769,18 @@ static void kfree_rcu_work(struct work_struct *work)* access is Ok. */ for (; head; head = next) { +unsigned long offset = (unsigned long )head->func; + next = head->next; -/* Could be possible to optimize with kfree_bulk in future */ +debug_rcu_head_unqueue (head); -__rcu_reclaim (rcu_state.name, head); +rcu_lock_acquire (&rcu_callback_map); +trace_rcu_invoke_kfree_callback (rcu_state.name, head, offset); + +/* Could be possible to optimize with kfree_bulk in future */ +kfree ((void *)head - offset); + +rcu_lock_release (&rcu_callback_map);cond_resched_tasks_rcu_qs (); } }@@ -2840,7 +2862,7 @@ static void kfree_rcu_monitor(struct work_struct *work)*/ void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) { -__call_rcu (head, func,1 ); +__call_rcu (head, func); }EXPORT_SYMBOL_GPL (kfree_call_rcu_nobatch);@@ -3094,7 +3116,7 @@ static void rcu_barrier_func(void *unused)debug_rcu_head_queue (&rdp->barrier_head);rcu_nocb_lock (rdp);WARN_ON_ONCE (!rcu_nocb_flush_bypass (rdp,NULL , jiffies)); -if (rcu_segcblist_entrain (&rdp->cblist, &rdp->barrier_head,0 )) { +if (rcu_segcblist_entrain (&rdp->cblist, &rdp->barrier_head)) {atomic_inc (&rcu_state.barrier_cpu_count); }else {debug_rcu_head_unqueue (&rdp->barrier_head);kernel/rcu/tree.h +0
-1@@ -183,7 +183,6 @@ struct rcu_data {bool rcu_urgent_qs;/* GP old need light quiescent state. */ bool rcu_forced_tick;/* Forced tick to provide QS. */ #ifdef CONFIG_RCU_FAST_NO_HZ -bool all_lazy;/* All CPU's CBs lazy at idle start? */ unsigned long last_accelerate;/* Last jiffy CBs were accelerated. */ unsigned long last_advance_all;/* Last jiffy CBs were all advanced. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ kernel/rcu/tree_plugin.h +0
-0@@ -1274,21 +1274,15 @@ static void rcu_prepare_for_idle(void)We now have only one preprocessor symbol (as opposed to the erroneous "three proprocessor symbols" in my original).
kernel/rcu/tree_plugin.h static void rcu_prepare_for_idle(void) · line 1278
Quoted 8 lines
kernel/rcu/tree_plugin.h +0
-3* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your * system. And if you are -that- concerned about energy efficiency, * just power the system down and be done with it! -* RCU_IDLE_LAZY_GP_DELAY gives the number of jiffies that a CPU is -* permitted to sleep in dyntick-idle mode with only lazy RCU -* callbacks pending. Setting this too high can OOM your system. * * The values below work well in practice. If future workloads require "value below works"
Quoted 28 lines
kernel/rcu/tree_plugin.h +1
-6* adjustment, they can be converted into kernel config parameters, though * making the state machine smarter might be a better option. */ #define RCU_IDLE_GP_DELAY 4/* Roughly one grace period. */ -#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ)/* Roughly six seconds. */ static int rcu_idle_gp_delay =RCU_IDLE_GP_DELAY ;module_param (rcu_idle_gp_delay, int,0644 ); -static int rcu_idle_lazy_gp_delay =RCU_IDLE_LAZY_GP_DELAY ; -module_param (rcu_idle_lazy_gp_delay, int,0644 );/* * Try to advance callbacks on the current CPU, but only if it has been @@ -1327,8 +1321,7 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void)/* * Allow the CPU to enter dyntick-idle mode unless it has callbacks ready * to invoke. If the CPU has callbacks, try to advance them. Tell the -* caller to set the timeout based on whether or not there are non-lazy -* callbacks. +* caller about what to set the timeout. * * The caller must have disabled interrupts. */ @@ -1354,25 +1347,19 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt) } rdp->last_accelerate = jiffies; -/* Request timer delay depending on laziness, and round. */ kernel/rcu/tree_plugin.h int rcu_needs_cpu(u64 basemono, u64 *nextevt) · line 1352
Quoted 10 lines
kernel/rcu/tree_plugin.h +3
-7- rdp->all_lazy = !rcu_segcblist_n_nonlazy_cbs (&rdp->cblist); -if (rdp->all_lazy) { - dj =round_jiffies (rcu_idle_lazy_gp_delay + jiffies) - jiffies; - }else { - dj =round_up (rcu_idle_gp_delay + jiffies, - rcu_idle_gp_delay) - jiffies; - } +/* Request timer and round. */ + dj =round_up (rcu_idle_gp_delay + jiffies, + rcu_idle_gp_delay) - jiffies;I bet that this now fits onto one line.
Quoted 58 lines
kernel/rcu/tree_plugin.h +5
-17+ *nextevt = basemono + dj *TICK_NSEC ;return 0 ; }/* -* Prepare a CPU for idle from an RCU perspective. The first major task -* is to sense whether nohz mode has been enabled or disabled via sysfs. -* The second major task is to check to see if a non-lazy callback has -* arrived at a CPU that previously had only lazy callbacks. The third -* major task is to accelerate (that is, assign grace-period numbers to) -* any recently arrived callbacks. +* Prepare a CPU for idle from an RCU perspective. The first major task is to +* sense whether nohz mode has been enabled or disabled via sysfs. The second +* major task is to accelerate (that is, assign grace-period numbers to) any +* recently arrived callbacks. * * The caller must have disabled interrupts. */ @@ -1398,17 +1385,6 @@ static void rcu_prepare_for_idle(void)if (!tne)return ; -/* -* If a non-lazy callback arrived at a CPU having only lazy -* callbacks, invoke RCU core for the side-effect of recalculating -* idle duration on re-entry to idle. -*/ -if (rdp->all_lazy &&rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)) { - rdp->all_lazy = false; -invoke_rcu_core (); -return ; - } -/* * If we have not yet accelerated this jiffy, accelerate all * callbacks on this CPU. kernel/rcu/tree_stall.h +2
-4@@ -263,11 +263,9 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) {struct rcu_data *rdp = &per_cpu (rcu_data, cpu); -sprintf (cp,"last_accelerate: %04lx/%04lx, Nonlazy posted: %c%c%c" , +sprintf (cp,"last_accelerate: %04lx/%04lx dyntick_enabled: %d" , rdp->last_accelerate &0xffff , jiffies &0xffff , -".l" [rdp->all_lazy], -".L" [!rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)], -".D" [!!rdp->tick_nohz_enabled_snap]); + !!rdp->tick_nohz_enabled_snap); }#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */ Signature · 1 line
2.23.0.187.g17f5b7556c-goog
PATCH 5/5 rcu: Remove kfree_call_rcu_nobatch()
In reply to a message not in the archiveNow that kfree_rcu() special casing have been removed from tree RCU, remove kfree_call_rcu_nobatch() since it is not needed.
--- .../admin-guide/kernel-parameters.txt | 4 --- include/linux/rcutiny.h | 5 --- include/linux/rcutree.h | 1 - kernel/rcu/rcuperf.c | 10 +----- kernel/rcu/tree.c | 33 ++++++++----------- 5 files changed, 14 insertions(+), 39 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +0 -4
@@ -3909,10 +3909,6 @@ Number of loops doing rcuperf.kfree_alloc_num number of allocations and frees. - rcuperf.kfree_no_batch= [KNL] - Use the non-batching (less efficient) version of kfree_rcu(). - This is useful for comparing with the batched version. - rcuperf.nreaders= [KNL] Set number of RCU readers. The value -1 selects N, where N is the number of CPUs. A value
include/linux/rcutiny.h +0 -5
@@ -39,11 +39,6 @@ static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)call_rcu (head, func); } -static inline void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) -{ -call_rcu (head, func); -} -void rcu_qs (void );static inline void rcu_softirq_qs (void )
include/linux/rcutree.h +0 -1
@@ -34,7 +34,6 @@ static inline void rcu_virt_note_context_switch(int cpu)void synchronize_rcu_expedited (void );void kfree_call_rcu (struct rcu_head *head,rcu_callback_t func); -void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func);void rcu_barrier (void );bool rcu_eqs_special_set (int cpu);
kernel/rcu/rcuperf.c +1 -9
@@ -593,7 +593,6 @@ rcu_perf_shutdown(void *arg)torture_param (int, kfree_nthreads,-1 ,"Number of threads running loops of kfree_rcu()." );torture_param (int, kfree_alloc_num,8000 ,"Number of allocations and frees done in an iteration." );torture_param (int, kfree_loops,10 ,"Number of loops doing kfree_alloc_num allocations and frees." ); -torture_param (int, kfree_no_batch,0 ,"Use the non-batching (slower) version of kfree_rcu()." );static struct task_struct **kfree_reader_tasks;static int kfree_nrealthreads;
@@ -632,14 +631,7 @@ kfree_perf_thread(void *arg)if (!alloc_ptr)return -ENOMEM ; -if (!kfree_no_batch) { -kfree_rcu (alloc_ptr, rh); - }else { -rcu_callback_t cb; - - cb = (rcu_callback_t )(unsigned long )offsetof(struct kfree_obj , rh); -kfree_call_rcu_nobatch (&(alloc_ptr->rh), cb); - } +kfree_rcu (alloc_ptr, rh); }cond_resched ();
kernel/rcu/tree.c +13 -20
@@ -2777,8 +2777,10 @@ static void kfree_rcu_work(struct work_struct *work)rcu_lock_acquire (&rcu_callback_map);trace_rcu_invoke_kfree_callback (rcu_state.name, head, offset); -/* Could be possible to optimize with kfree_bulk in future */ -kfree ((void *)head - offset); +if (!WARN_ON_ONCE (!__is_kfree_rcu_offset (offset))) { +/* Could be optimized with kfree_bulk() in future. */ +kfree ((void *)head - offset); + }rcu_lock_release (&rcu_callback_map);cond_resched_tasks_rcu_qs ();
@@ -2856,16 +2858,6 @@ static void kfree_rcu_monitor(struct work_struct *work)spin_unlock_irqrestore (&krcp->lock, flags); } -/* -* This version of kfree_call_rcu does not do batching of kfree_rcu() requests. -* Used only by rcuperf torture test for comparison with kfree_rcu_batch(). -*/ -void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) -{ -__call_rcu (head, func); -} -EXPORT_SYMBOL_GPL (kfree_call_rcu_nobatch); -/* * Queue a request for lazy invocation of kfree() after a grace period. *
@@ -2885,12 +2877,6 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)unsigned long flags;struct kfree_rcu_cpu *krcp; -/* kfree_call_rcu() batching requires timers to be up. If the scheduler -* is not yet up, just skip batching and do the non-batched version. -*/ -if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING ) -return kfree_call_rcu_nobatch (head, func); -if (debug_rcu_head_queue (head)) {/* Probable double kfree_rcu() */ WARN_ONCE(1, "kfree_call_rcu(): Double-freed call. rcu_head %p\n",
@@ -2909,8 +2895,15 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) krcp->head = head;/* Schedule monitor for timely drain after KFREE_DRAIN_JIFFIES. */ -if (!xchg (&krcp->monitor_todo, true)) -schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); +if (!xchg (&krcp->monitor_todo, true)) { +/* Scheduling the monitor requires scheduler/timers to be up, +* if it is not, just skip it. An eventual kfree_rcu() will +* kick it again. +*/ +if ((rcu_scheduler_active ==RCU_SCHEDULER_RUNNING )) { +schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); + } + }spin_unlock (&krcp->lock);local_irq_restore (flags);
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Paul E. McKenney
maintainerRe: [PATCH 5/5] rcu: Remove kfree_call_rcu_nobatch()
Quoted 4 lines · Joel Fernandes (Google)
Now that
kfree_rcu()special casing have been removed from tree RCU, removekfree_call_rcu_nobatch()since it is not needed.Signed-off-by Joel Fernandes (Google) <[email protected]>
Now -this- one qualifies as a nice negative delta! ;-)
A few things below, please fix in next version.
Thanx, Paul
Quoted 84 lines
--- .../admin-guide/kernel-parameters.txt | 4 --- include/linux/rcutiny.h | 5 --- include/linux/rcutree.h | 1 - kernel/rcu/rcuperf.c | 10 +----- kernel/rcu/tree.c | 33 ++++++++----------- 5 files changed, 14 insertions(+), 39 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-4@@ -3909,10 +3909,6 @@ Number of loops doing rcuperf.kfree_alloc_num number of allocations and frees. - rcuperf.kfree_no_batch= [KNL] - Use the non-batching (less efficient) version of kfree_rcu(). - This is useful for comparing with the batched version. - rcuperf.nreaders= [KNL] Set number of RCU readers. The value -1 selects N, where N is the number of CPUs. A valueinclude/linux/rcutiny.h +0
-5@@ -39,11 +39,6 @@ static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)call_rcu (head, func); } -static inline void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) -{ -call_rcu (head, func); -} -void rcu_qs (void );static inline void rcu_softirq_qs (void )include/linux/rcutree.h +0
-1@@ -34,7 +34,6 @@ static inline void rcu_virt_note_context_switch(int cpu)void synchronize_rcu_expedited (void );void kfree_call_rcu (struct rcu_head *head,rcu_callback_t func); -void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func);void rcu_barrier (void );bool rcu_eqs_special_set (int cpu);kernel/rcu/rcuperf.c +1
-9@@ -593,7 +593,6 @@ rcu_perf_shutdown(void *arg)torture_param (int, kfree_nthreads,-1 ,"Number of threads running loops of kfree_rcu()." );torture_param (int, kfree_alloc_num,8000 ,"Number of allocations and frees done in an iteration." );torture_param (int, kfree_loops,10 ,"Number of loops doing kfree_alloc_num allocations and frees." ); -torture_param (int, kfree_no_batch,0 ,"Use the non-batching (slower) version of kfree_rcu()." );static struct task_struct **kfree_reader_tasks;static int kfree_nrealthreads;@@ -632,14 +631,7 @@ kfree_perf_thread(void *arg)if (!alloc_ptr)return -ENOMEM ; -if (!kfree_no_batch) { -kfree_rcu (alloc_ptr, rh); - }else { -rcu_callback_t cb; - - cb = (rcu_callback_t )(unsigned long )offsetof(struct kfree_obj , rh); -kfree_call_rcu_nobatch (&(alloc_ptr->rh), cb); - } +kfree_rcu (alloc_ptr, rh); }cond_resched ();kernel/rcu/tree.c +0
-0
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2783
Quoted 9 lines
kernel/rcu/tree.c +4
-2@@ -2777,8 +2777,10 @@ static void kfree_rcu_work(struct work_struct *work)rcu_lock_acquire (&rcu_callback_map);trace_rcu_invoke_kfree_callback (rcu_state.name, head, offset); -/* Could be possible to optimize with kfree_bulk in future */ -kfree ((void *)head - offset); +if (!WARN_ON_ONCE (!__is_kfree_rcu_offset (offset))) { +/* Could be optimized with kfree_bulk() in future. */ +kfree ((void *)head - offset); + }
This really needs to be in the previous patch until such time as Tiny RCU no longer needs the restriction.
Quoted 37 lines
kernel/rcu/tree.c +0
-17rcu_lock_release (&rcu_callback_map);cond_resched_tasks_rcu_qs ();@@ -2856,16 +2858,6 @@ static void kfree_rcu_monitor(struct work_struct *work)spin_unlock_irqrestore (&krcp->lock, flags); } -/* -* This version of kfree_call_rcu does not do batching of kfree_rcu() requests. -* Used only by rcuperf torture test for comparison with kfree_rcu_batch(). -*/ -void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) -{ -__call_rcu (head, func); -} -EXPORT_SYMBOL_GPL (kfree_call_rcu_nobatch); -/* * Queue a request for lazy invocation of kfree() after a grace period. * @@ -2885,12 +2877,6 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)unsigned long flags;struct kfree_rcu_cpu *krcp; -/* kfree_call_rcu() batching requires timers to be up. If the scheduler -* is not yet up, just skip batching and do the non-batched version. -*/ -if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING ) -return kfree_call_rcu_nobatch (head, func); -if (debug_rcu_head_queue (head)) {/* Probable double kfree_rcu() */ WARN_ONCE(1, "kfree_call_rcu(): Double-freed call. rcu_head %p\n",@@ -2909,8 +2895,15 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) krcp->head = head;/* Schedule monitor for timely drain after KFREE_DRAIN_JIFFIES. */ -if (!xchg (&krcp->monitor_todo, true))
kernel/rcu/tree.c void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) · line 2906
Quoted 10 lines
kernel/rcu/tree.c +9
-1-schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); +if (!xchg (&krcp->monitor_todo, true)) { +/* Scheduling the monitor requires scheduler/timers to be up, +* if it is not, just skip it. An eventual kfree_rcu() will +* kick it again. +*/ +if ((rcu_scheduler_active ==RCU_SCHEDULER_RUNNING )) { +schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); + } + }
And this also needs to be in an earlier patch. Bisectability and all that!
Are we really guaranteed that there will be an eventual kfree_rcu()? More of a worry for Tiny RCU than for Tree RCU, but still could be annoying for someone trying to debug a memory leak.
Thanx, Paul
Quoted 5 lines
kernel/rcu/tree.c +0
-0spin_unlock (&krcp->lock);local_irq_restore (flags);Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Re: [PATCH 5/5] rcu: Remove kfree_call_rcu_nobatch()
Hi Paul,
I think this is the only contentious patch preventing my resend of the series, let me know what you think, I replied below:
Quoted 1 line · Paul E. McKenney
On Tue, Aug 27, 2019 at 03:01:59PM -0400, Joel Fernandes (Google) wrote:
[snip]
Quoted 8 lines
Quoted 8 lines
kernel/rcu/tree.c +0
-0@@ -2777,8 +2777,10 @@ static void kfree_rcu_work(struct work_struct *work)rcu_lock_acquire (&rcu_callback_map);trace_rcu_invoke_kfree_callback (rcu_state.name, head, offset);
Quoted 9 lines
kernel/rcu/tree.c static void kfree_rcu_work(struct work_struct *work) · line 2783
Quoted 6 lines
kernel/rcu/tree.c +4
-2-/* Could be possible to optimize with kfree_bulk in future */ -kfree ((void *)head - offset); +if (!WARN_ON_ONCE (!__is_kfree_rcu_offset (offset))) { +/* Could be optimized with kfree_bulk() in future. */ +kfree ((void *)head - offset); + }This really needs to be in the previous patch until such time as Tiny RCU no longer needs the restriction.
I was only going by whatever is already committed to the -rcu dev branch. The series is based on the -dev branch.
The original patch adding the kfree_rcu() batching is already merged into the -rcu dev branch (that version just had 1 list, this series adds multiple lists).
In the above diff, I just added the WARN_ON_ONCE() as extra checking for tree RCU kfree batching. It has nothing to do with tiny RCU per-se. Should I submit the WARN_ON_ONCE() as a separate patch then?
To prevent confusion, could you let me know if I am supposed to submitting patches against a branch other than the dev branch?
Quoted 50 lines
Quoted 37 lines
kernel/rcu/tree.c +0
-17rcu_lock_release (&rcu_callback_map);cond_resched_tasks_rcu_qs ();@@ -2856,16 +2858,6 @@ static void kfree_rcu_monitor(struct work_struct *work)spin_unlock_irqrestore (&krcp->lock, flags); } -/* -* This version of kfree_call_rcu does not do batching of kfree_rcu() requests. -* Used only by rcuperf torture test for comparison with kfree_rcu_batch(). -*/ -void kfree_call_rcu_nobatch (struct rcu_head *head,rcu_callback_t func) -{ -__call_rcu (head, func); -} -EXPORT_SYMBOL_GPL (kfree_call_rcu_nobatch); -/* * Queue a request for lazy invocation of kfree() after a grace period. * @@ -2885,12 +2877,6 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)unsigned long flags;struct kfree_rcu_cpu *krcp; -/* kfree_call_rcu() batching requires timers to be up. If the scheduler -* is not yet up, just skip batching and do the non-batched version. -*/ -if (rcu_scheduler_active !=RCU_SCHEDULER_RUNNING ) -return kfree_call_rcu_nobatch (head, func); -if (debug_rcu_head_queue (head)) {/* Probable double kfree_rcu() */ WARN_ONCE(1, "kfree_call_rcu(): Double-freed call. rcu_head %p\n",@@ -2909,8 +2895,15 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) krcp->head = head;/* Schedule monitor for timely drain after KFREE_DRAIN_JIFFIES. */ -if (!xchg (&krcp->monitor_todo, true))kernel/rcu/tree.c void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) · line 2906
Quoted 10 lines
kernel/rcu/tree.c +9
-1-schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); +if (!xchg (&krcp->monitor_todo, true)) { +/* Scheduling the monitor requires scheduler/timers to be up, +* if it is not, just skip it. An eventual kfree_rcu() will +* kick it again. +*/ +if ((rcu_scheduler_active ==RCU_SCHEDULER_RUNNING )) { +schedule_delayed_work (&krcp->monitor_work,KFREE_DRAIN_JIFFIES ); + } + }And this also needs to be in an earlier patch. Bisectability and all that!
Quoted 3 lines
Are we really guaranteed that there will be an eventual
kfree_rcu()? More of a worry for Tiny RCU than for Tree RCU, but still could be annoying for someone trying to debug a memory leak.
Same comment as above, the original patch adding the schedule_delayed_work() is already merged into the -dev branch. This series is based on top of that. The reason I had to rearrange &krcp->monitor_todo code above is because we no longer have kfree_rcu_no_batch() which this patch removes.
thanks,
- Joel