RCU updates
Subsystems (6)
cover RCU updates
Here are the patches that I have been testing that should help some of the latency and OOM issues (file limit) that we had discussed in the past. If the patchset looks ok,
we should queue them up in -mm for some testing before merging. I have lightly tested the patchset on both ppc64 and x86_64 using ltp, dbench etc.
Update since the last time I published - file counting stuff uses percpu_counter.
Thanks
Dipankar
Re: [PATCH 2/2] fix file counting
In reply to Dipankar SarmaI have benchmarked this on an x86_64 NUMA system and see no significant performance difference on kernbench. Tested on both x86_64 and powerpc.
The way we do file struct accounting is not very suitable for batched freeing. For scalability reasons, file accounting was constructor/destructor based. This meant that nr_files was decremented only when
the object was removed from the slab cache. This is
susceptible to slab fragmentation. With RCU based file structure, consequent batched freeing and a test program like Serge's,
we just speed this up and end up with a very fragmented slab -
llm22:~ # cat /proc/sys/fs/file-nr
587730 0 758844
At the same time, I see only a 2000+ objects in filp cache. The following patch I fixes this problem.
This patch changes the file counting by removing the filp_count_lock. Instead we use a separate percpu counter, nr_files, for now and all accesses to it are through get_nr_files() api. In the sysctl handler for nr_files, we populate files_stat.nr_files before returning to user.
Counting files as an when they are created and destroyed (as opposed to inside slab) allows us to correctly count open files with RCU.
---
fs/dcache.c | 2 - fs/file_table.c | 80 +++++++++++++++++++++++++++++++-------------------- include/linux/file.h | 2 - include/linux/fs.h | 2 + kernel/sysctl.c | 5 ++- net/unix/af_unix.c | 2 - 6 files changed, 57 insertions(+), 36 deletions(-)
fs/dcache.c renamed from fs/dcache.c~fix-file-counting +1 -1
@@ -1736,7 +1736,7 @@ void __init vfs_caches_init(unsigned lon SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL, NULL); filp_cachep =kmem_cache_create ("filp" ,sizeof (struct file ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , filp_ctor, filp_dtor); +SLAB_HWCACHE_ALIGN |SLAB_PANIC ,NULL ,NULL );dcache_init (mempages);inode_init (mempages);
fs/file_table.c renamed from fs/file_table.c~fix-file-counting +49 -31
134 lines not on this page · raw message
include/linux/file.h renamed from include/linux/file.h~fix-file-counting +0 -2
@@ -60,8 +60,6 @@ extern void put_filp(struct file *);extern int get_unused_fd (void );extern void FASTCALL (put_unused_fd (unsigned int fd));struct kmem_cache ; -extern void filp_ctor (void * objp,struct kmem_cache *cachep,unsigned long cflags); -extern void filp_dtor (void * objp,struct kmem_cache *cachep,unsigned long dflags);extern struct file **alloc_fd_array (int );extern void free_fd_array (struct file **,int );
include/linux/fs.h renamed from include/linux/fs.h~fix-file-counting +2 -0
@@ -35,6 +35,8 @@ struct files_stat_struct {int max_files;/* tunable */ };extern struct files_stat_struct files_stat; +extern int get_nr_files (void ); +extern int get_max_files (void );struct inodes_stat_t {int nr_inodes;
kernel/sysctl.c renamed from kernel/sysctl.c~fix-file-counting +4 -1
@@ -52,6 +52,9 @@#include <linux/nfs_fs.h> #endif +extern int proc_nr_files (ctl_table *table,int write,struct file *filp, +void __user *buffer,size_t *lenp,loff_t *ppos); +#if defined(CONFIG_SYSCTL )/* External variables not in a header file. */
@@ -921,7 +924,7 @@ static ctl_table fs_table[] = { .data = &files_stat, .maxlen = 3*sizeof(int), .mode = 0444, - .proc_handler = &proc_dointvec, + .proc_handler = &proc_nr_files, }, { .ctl_name = FS_MAXFILE,
net/unix/af_unix.c renamed from net/unix/af_unix.c~fix-file-counting +1 -1
@@ -547,7 +547,7 @@ static struct sock * unix_create1(structstruct sock *sk =NULL ;struct unix_sock *u; -if (atomic_read (&unix_nr_socks) >=2 *files_stat.max_files) +if (atomic_read (&unix_nr_socks) >=2 *get_max_files ()) goto out; sk =sk_alloc (PF_UNIX ,GFP_KERNEL , &unix_proto,1 );
_
Re: [PATCH 1/2] rcu batch tuning
In reply to Dipankar SarmaThis patch adds new tunables for RCU queue and finished batches. There are two types of controls - number of completed RCU updates invoked in a batch (blimit) and monitoring for high rate of incoming RCUs on a cpu (qhimark, qlowmark). By default,
the per-cpu batch limit is set to a small value. If
the input RCU rate exceeds the high watermark, we do two things - force quiescent state on all cpus and set the batch limit
of the CPU to INTMAX. Setting batch limit to INTMAX forces all finished RCUs to be processed in one shot. If we have more than INTMAX RCUs queued up, then we have bigger problems anyway. Once the incoming queued RCUs fall below the low watermark, the batch limit is set to the default.
---
include/linux/rcupdate.h | 6 +++ kernel/rcupdate.c | 76 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 63 insertions(+), 19 deletions(-)
include/linux/rcupdate.h renamed from include/linux/rcupdate.h~rcu-batch-tuning +5 -1
@@ -98,13 +98,17 @@ struct rcu_data {long batch;/* Batch # for current RCU batch */ struct rcu_head *nxtlist;struct rcu_head **nxttail; -long count;/* # of queued items */ +long qlen;/* # of queued callbacks */ struct rcu_head *curlist;struct rcu_head **curtail;struct rcu_head *donelist;struct rcu_head **donetail; +long blimit;/* Upper limit on a processed batch */ int cpu;struct rcu_head barrier; +#ifdef CONFIG_SMP +long last_rs_qlen;/* qlen during the last resched */ +#endif }; DECLARE_PER_CPU(struct rcu_data, rcu_data);
kernel/rcupdate.c renamed from kernel/rcupdate.c~rcu-batch-tuning +58 -18
117 lines not on this page · raw message
_
Re: [PATCH 1/2] rcu batch tuning
Andrew,
The earlier version of this patch was against 2.6.16-rc3 and didn't apply cleanly to 2.6.16-rc3-mm1. This version applies cleanly to 2.6.16-rc3-mm1.
This patch adds new tunables for RCU queue and finished batches. There are two types of controls - number of completed RCU updates invoked in a batch (blimit) and monitoring for high rate of incoming RCUs on a cpu (qhimark, qlowmark). By default,
the per-cpu batch limit is set to a small value. If
the input RCU rate exceeds the high watermark, we do two things - force quiescent state on all cpus and set the batch limit
of the CPU to INTMAX. Setting batch limit to INTMAX forces all finished RCUs to be processed in one shot. If we have more than INTMAX RCUs queued up, then we have bigger problems anyway. Once the incoming queued RCUs fall below the low watermark, the batch limit is set to the default.
---
include/linux/rcupdate.h | 6 +++ kernel/rcupdate.c | 74 ++++++++++++++++++++++++++++++++++++----------- 2 files changed, 63 insertions(+), 17 deletions(-)
include/linux/rcupdate.h renamed from include/linux/rcupdate.h~rcu-batch-tuning +5 -1
@@ -98,13 +98,17 @@ struct rcu_data {long batch;/* Batch # for current RCU batch */ struct rcu_head *nxtlist;struct rcu_head **nxttail; -long count;/* # of queued items */ +long qlen;/* # of queued callbacks */ struct rcu_head *curlist;struct rcu_head **curtail;struct rcu_head *donelist;struct rcu_head **donetail; +long blimit;/* Upper limit on a processed batch */ int cpu;struct rcu_head barrier; +#ifdef CONFIG_SMP +long last_rs_qlen;/* qlen during the last resched */ +#endif }; DECLARE_PER_CPU(struct rcu_data, rcu_data);
kernel/rcupdate.c renamed from kernel/rcupdate.c~rcu-batch-tuning +58 -16
116 lines not on this page · raw message
_
Re: [PATCH 1/2] rcu batch tuning
In reply to Dipankar SarmaQuoted 8 lines · Dipankar Sarma
-module_param(maxbatch, int, 0);
+module_param(blimit, int, 0);
+module_param(qhimark, int, 0);
+module_param(qlowmark, int, 0);
+#ifdefCONFIG_SMP
+module_param(rsinterval, int, 0);
+#endif
It's a bit unusual to add boot-time tunables via module_param, but there's no law against it.
But you do get arrested for not adding them to Documentation/kernel-parameters.txt. That's if you think they're permanent (I hope they aren't). If they are, they'll probably need a more extensive description than kernel-parameters.txt entries normally provide.
Re: [PATCH 2/2] fix file counting
In reply to Dipankar SarmaQuoted 19 lines · Dipankar Sarma
I have benchmarked this on an
x86_64NUMA system and see no
significant performance difference on kernbench. Tested on bothx86_64and powerpc.The way we do file struct accounting is not very suitable for batched freeing. For scalability reasons, file accounting was constructor/destructor based. This meant that
nr_fileswas decremented only when
the object was removed from the slab cache. This is
susceptible to slab fragmentation. With RCU based file structure, consequent batched freeing and a test program like Serge's,
we just speed this up and end up with a very fragmented slab -llm22:~ # cat
/proc/sys/fs/file-nr
587730 0 758844At the same time, I see only a 2000+ objects in filp cache. The following patch I fixes this problem.
Quoted 8 lines
This patch changes the file counting by removing the
filp_count_lock. Instead we use a separate percpu counter,nr_files, for now and all accesses to it are throughget_nr_files()api. In the sysctl handler fornr_files, we populate files_stat.nr_files before returning to user.Counting files as an when they are created and destroyed (as opposed to inside slab) allows us to correctly count open files with RCU.
Fair enough.
What do you think of these changes?
From: Andrew Morton <[email protected]>
- Nuke the blank line between "}" and EXPORT_SYMBOL(). That's never seemed pointful to me.
- Make the get_max_files export use _GPL - only unix.ko uses it.
- Use -1' in the arg to percpu_counter_mod() rather than -1L'. The compiler will dtrt and we shouldn't be peering inside percpu_counter internals here anyway.
- Scrub that - use percpu_counter_dec() and percpu_counter_inc().
- percpu_counters can be inaccurate on big SMP. Before we actually fail a get_empty_filp() attempt, use the (new in -mm) expensive percpu_counter_sum() to check whether we're really over the limit.
- Make get_nr_files() static. Which is just as well - any callers might want the percpu_counter_sum() treatment. In which case we'd be better off exporting some
bool are_files_over_limit(how_many_i_want);
API.
---
fs/file_table.c | 21 +++++++++++++-------- include/linux/fs.h | 1 - 2 files changed, 13 insertions(+), 9 deletions(-)
fs/file_table.c renamed from fs/file_table.c~fix-file-counting-fixes +13 -8
@@ -22,6 +22,7 @@#include <linux/fsnotify.h> #include <linux/sysctl.h> #include <linux/percpu_counter.h> +#include <asm/atomic.h> /* sysctl tunables... */
@@ -42,14 +43,14 @@ static inline void file_free_rcu(structstatic inline void file_free (struct file *f) { -percpu_counter_mod (&nr_files,-1L ); +percpu_counter_dec (&nr_files);call_rcu (&f->f_u.fu_rcuhead, file_free_rcu); }/* * Return the total number of open files in the system */ -int get_nr_files (void ) +static int get_nr_files (void ) {return percpu_counter_read_positive (&nr_files); }
@@ -61,8 +62,7 @@ int get_max_files(void) {return files_stat.max_files; } - -EXPORT_SYMBOL (get_max_files); +EXPORT_SYMBOL_GPL (get_max_files);/* * Handle nr_files sysctl
@@ -95,15 +95,20 @@ struct file *get_empty_filp(void)/* * Privileged users can go above max_files */ -if (get_nr_files () >= files_stat.max_files && - !capable (CAP_SYS_ADMIN )) - goto over; +if (get_nr_files () >= files_stat.max_files && !capable (CAP_SYS_ADMIN )) { +/* +* percpu_counters are inaccurate. Do an expensive check before +* we go and fail. +*/ +if (percpu_counter_sum (&nr_files) >= files_stat.max_files) + goto over; + } f =kmem_cache_alloc (filp_cachep,GFP_KERNEL );if (f ==NULL ) goto fail; -percpu_counter_mod (&nr_files,1L ); +percpu_counter_inc (&nr_files);memset (f,0 ,sizeof (*f));if (security_file_alloc (f)) goto fail_sec;
include/linux/fs.h renamed from include/linux/fs.h~fix-file-counting-fixes +0 -1
@@ -35,7 +35,6 @@ struct files_stat_struct {int max_files;/* tunable */ };extern struct files_stat_struct files_stat; -extern int get_nr_files (void );extern int get_max_files (void );struct inodes_stat_t {
_
Re: [PATCH 1/2] rcu batch tuning
In reply to Andrew MortonQuoted 10 lines · Andrew Morton
Quoted 8 lines · Dipankar Sarma
-module_param(maxbatch, int, 0);
+module_param(blimit, int, 0);
+module_param(qhimark, int, 0);
+module_param(qlowmark, int, 0);
+#ifdefCONFIG_SMP
+module_param(rsinterval, int, 0);
+#endif
Quoted 7 lines
It's a bit unusual to add boot-time tunables via
module_param, but there's no law against it.But you do get arrested for not adding them to
Documentation/kernel-parameters.txt. That's if you think they're permanent (I hope they aren't). If they are, they'll probably need a more extensive description than kernel-parameters.txt entries normally provide.
I hope that we will not need that many tunables eventually. But my theory has been that with widespread use and experiments with the tunables in the event of OOM and latency problems will allow us to figure out what kind of automatic tuning really works.
Regardless, I think there should be documentation. I will send a documentation patch.
Thanks
Dipankar
Re: [PATCH 2/2] fix file counting
In reply to Andrew MortonQuoted 8 lines · Andrew Morton
Dipankar Sarma <[email protected]> wrote:
Fair enough.What do you think of these changes?
- Nuke the blank line between "}" and
EXPORT_SYMBOL(). That's never seemed pointful to me.
Sounds good.
Quoted 2 lines
- Make the
get_max_filesexport use_GPL- only unix.ko uses it.
Always good. I just didn't want to be the one :)
Quoted 5 lines
- Use
-1' in the arg to percpu_counter_mod() rather than-1L'. The compiler will dtrt and we shouldn't be peering insidepercpu_counterinternals here anyway.- Scrub that - use
percpu_counter_dec()andpercpu_counter_inc().
Gah! I missed those APIs. Makes perfect sense.
Quoted 7 lines
-
percpu_counterscan be inaccurate on big SMP. Before we actually fail aget_empty_filp()attempt, use the (new in -mm) expensivepercpu_counter_sum()to check whether we're really over the limit.fs/file_table.c renamed from fs/file_table.c~fix-file-counting-fixes +0
-1- if (get_nr_files() >= files_stat.max_files &&
fs/file_table.c struct file *get_empty_filp(void) · line 105
Quoted 10 lines
fs/file_table.c renamed from fs/file_table.c~fix-file-counting-fixes +8
-2- !capable(CAP_SYS_ADMIN)) - goto over; +if (get_nr_files () >= files_stat.max_files && !capable (CAP_SYS_ADMIN )) { +/* +* percpu_counters are inaccurate. Do an expensive check before +* we go and fail. +*/ +if (percpu_counter_sum (&nr_files) >= files_stat.max_files) + goto over; + }
Slight optimization -
if (get_nr_files() >= files_stat.max_files) {
if (capable(CAP_SYS_ADMIN)) {
/*
* percpu_counters are inaccurate. Do an expensive check before
* we go and fail.
*/
if (percpu_counter_sum(&nr_files) >=
files_stat.max_files)
goto over;
} else
goto over;
}Quoted 7 lines
- Make
get_nr_files()static. Which is just as well - any callers might want thepercpu_counter_sum()treatment. In which case we'd be better off exporting somebool are_files_over_limit(how_many_i_want);
Well, as of now there is none (xfs use was removed a few months ago). So, I removed the EXPORT_SYMBOL for get_nr_files() and forgot to make it static. Your patch does the right thing.
Thanks
Dipankar
Re: [PATCH 2/2] fix file counting
Quoted 17 lines · Dipankar Sarma
fs/file_table.c struct file *get_empty_filp(void) · line 105
Quoted 12 lines
fs/file_table.c renamed from fs/file_table.c~fix-file-counting-fixes +8
-3-if (get_nr_files () >= files_stat.max_files && - !capable (CAP_SYS_ADMIN )) - goto over; +if (get_nr_files () >= files_stat.max_files && !capable (CAP_SYS_ADMIN )) { +/* +* percpu_counters are inaccurate. Do an expensive check before +* we go and fail. +*/ +if (percpu_counter_sum (&nr_files) >= files_stat.max_files) + goto over; + }Slight optimization -
(quoted diff) +0
-0if (get_nr_files() >= files_stat.max_files) { if (capable(CAP_SYS_ADMIN)) {
Review
Quoted 10 lines
(quoted diff) +0
-0/* * percpu_counters are inaccurate. Do an expensive check before * we go and fail. */ if (percpu_counter_sum(&nr_files) >= files_stat.max_files) goto over; } else goto over; }
That changes the behaviour for root. Maybe you meant !capable(), but that still changes the behaviour. I'm all confused.
Re: [PATCH 2/2] fix file counting
Quoted 18 lines · Andrew Morton
Quoted 6 lines · Dipankar Sarma
Slight optimization -
(quoted diff) +0
-0if (get_nr_files() >= files_stat.max_files) { if (capable(CAP_SYS_ADMIN)) {Review
Quoted 10 lines
(quoted diff) +0
-0/* * percpu_counters are inaccurate. Do an expensive check before * we go and fail. */ if (percpu_counter_sum(&nr_files) >= files_stat.max_files) goto over; } else goto over; }
Quoted 2 lines
That changes the behaviour for root. Maybe you meant !capable(), but that still changes the behaviour. I'm all confused.
Hmm.. on second thoughts, there is no harm doing the expensive check for both priviledged and non-priviledged user. It will correctly allow non-priviledged users to create the new file provided the fast-path percpu counter value returned was greater than the slow path per-cpu counter value.
Just ignore my comment in the previous mail.
Thanks
Dipankar
Re: [PATCH 2/2] fix file counting
Quoted 20 lines · Dipankar Sarma
Quoted 18 lines · Andrew Morton
Quoted 16 lines · Dipankar Sarma
Slight optimization -
(quoted diff) +0
-0if (get_nr_files() >= files_stat.max_files) { if (capable(CAP_SYS_ADMIN)) { /* * percpu_counters are inaccurate. Do an expensive check before * we go and fail. */ if (percpu_counter_sum(&nr_files) >= files_stat.max_files) goto over; } else goto over; }
Quoted 8 lines
Quoted 2 lines
That changes the behaviour for root. Maybe you meant !capable(), but that still changes the behaviour. I'm all confused.
Hmm.. on second thoughts, there is no harm doing the expensive check for both priviledged and non-priviledged user. It will correctly allow non-priviledged users to create the new file provided the fast-path percpu counter value returned was greater than the slow path per-cpu counter value.
Look closer ;)
if (get_nr_files() >= files_stat.max_files && !capable(CAP_SYS_ADMIN)) {
/*
* percpu_counters are inaccurate. Do an expensive check before
* we go and fail.
*/
if (percpu_counter_sum(&nr_files) >= files_stat.max_files)
goto over;
}Re: [PATCH 2/2] fix file counting
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Dipankar Sarma
On Sat, Feb 18, 2006 at 01:45:29AM -0800, Andrew Morton wrote:
Look closer ;)
Quoted 8 lines
if (get_nr_files() >= files_stat.max_files && !capable(CAP_SYS_ADMIN)) {
/*
* percpu_counters are inaccurate. Do an expensive check before
* we go and fail.
*/
if (percpu_counter_sum(&nr_files) >= files_stat.max_files)
goto over;
}Ah, that check is not relevant for priviledged users anyway. That means I should just continue doing where my mind is at the moment - running the weekend errands :) And not look at code.
Thanks
Dipankar
Re: [PATCH 2/2] fix file counting
In reply to Dipankar SarmaQuoted 1 line
Quoted 1 line
- Make the
get_max_filesexport use_GPL- only unix.ko uses it.
The real question is, does af_unix really need to allow beeing built modular? It's quite different from other network protocol and deeply tied to the kernel due to things like descriptor passing or using the filesystem namespace. I already had to export another symbol that really should be internal just for it, and if one module acquires lots of such hacks it's usually a bad sign..
Re: [PATCH 2/2] fix file counting
Quoted 8 lines · Christoph Hellwig
Quoted 1 line
Quoted 1 line
- Make the
get_max_filesexport use_GPL- only unix.ko uses it.The real question is, does
af_unixreally need to allow beeing built modular? It's quite different from other network protocol and deeply tied to the kernel due to things like descriptor passing or using the filesystem namespace. I already had to export another symbol that really should be internal just for it, and if one module acquires lots of such hacks it's usually a bad sign..
in 2.4 the answer would have been simple; modutils back then used AF_UNIX stuff before it could load modules, so modular was in practice impossible.
Anyway I'd agree with making this non-modular... NOBODY will use this as a module, or if they do loading it somehow is the very first thing done. You just can't live without this, so making it a module is non-sensical.
PATCH 1/1 make UNIX a bool
Quoted 10 lines · Arjan van de Ven
Quoted 8 lines · Christoph Hellwig
Quoted 1 line
Quoted 1 line
- Make the
get_max_filesexport use_GPL- only unix.ko uses it.The real question is, does
af_unixreally need to allow beeing built modular? It's quite different from other network protocol and deeply tied to the kernel due to things like descriptor passing or using the filesystem namespace. I already had to export another symbol that really should be internal just for it, and if one module acquires lots of such hacks it's usually a bad sign..
Quoted 7 lines
in 2.4 the answer would have been simple; modutils back then used
AF_UNIXstuff before it could load modules, so modular was in practice impossible.Anyway I'd agree with making this non-modular... NOBODY will use this as a module, or if they do loading it somehow is the very first thing done. You just can't live without this, so making it a module is non-sensical.
So let's send a patch. ;-)
cu
Adrian
<-- snip -->
CONFIG_UNIX=m doesn't make much sense.
net/unix/Kconfig renamed from net/unix/Kconfig.old +1 -1
@@ -3,7 +3,7 @@ # config UNIX - tristate "Unix domain sockets" + bool "Unix domain sockets" ---help--- If you say Y here, you will include support for Unix domain sockets; sockets are the standard Unix mechanism for establishing and