brsem: [RFC] big reader semaphore
Subsystems (9)
cover brsem: [RFC] big reader semaphore
Hello, guys.
This patchset implements brsem - big reader semaphore. This is
similar to the late big reader lock in its concept. Reader side tries to do as little as possible while writer side takes over all the overhead. brsem is to be used for cases where...
a. writer operation is *extremely* rare compared to read ops
b. both readers and writers should be able to sleep
c. stale data or reader-side retry is impossible or impractical
Device or subsystem hotplug/unplug synchronization fits very well to
above criteria. Plugging and unplugging occur very rarely but they still need stringent synchronization.
brsem is implemented while trying to solve the following race
condition in the vfs layer.
super->s_umount rwsem proctects filesystem unmounts and remounts
against other operations. When remounting a rw filesystem ro, after write-locking s_umount, do_remount_sb() is invoked, which scans all open files and either refuses to remount or bang all write-open files depending on force argument. After that, the filesystem is remounted ro.
The problem is that a file can be opened rw after all open files are
scanned but before the filesystem is remounted ro. This can be easily exposed by adding msleep(2000) between open file scan and remount_fs() and doing the following in the user space. An ext2 fs is mounted on tmp.
# mount -o remount,ro tmp& sleep 1; cat >> tmp/asdf
This leaves us with a write-opened file on a ro filesystem. ext2
happily writes data to disk. I've only verified open(2) but this race probably exists for other file operations, too.
I'll post a patch to add 2s sleep during remounting and test results
in a reply to this mail.
The solution is to make permission check and the rest of open atomic
against remounting by read-locking s_umount rwsem. This probably should be done for most file operations including write(2) (maybe even when committing dirty pages to fs for rw-mmapped files in 'force' case). This is an expensive overhead to pay for such rare cases and seqlock / rcupdate cannot be used here.
So, brsem is implemented. Read locking and unlocking involve only
local_bh_disable/enable, a few local arithmetics and conditionals - no atomic ops, no shared word, no memory barrier. Writer operations are *very* expensive. They have to issue workqueue works to each cpu using smp_call_function and wait for them.
This patchset also converts cpucontrol semaphore which protects cpu
hotplug/unplugging events, which obviously occur very rarely.
[ Start of patch descriptions ]
01_brsem_implement_brsem.patch
: implement big reader semaphore
This patch implements big reader semaphore - a rwsem with very cheap reader-side operations and very expensive writer-side operations. For details, please read comments at the top of kern/brsem.c.
02_brsem_convert-s_umount-to-brsem.patch
: convert super_block->s_umount to brsem Convert super_block->s_umount from rwsem to brsem.
03_brsem_fix-remount-open-race.patch
: fix ro-remount <-> open race condition
A file can be opened rw while ro-remounting is in progress resulting in open rw file on ro mounted filesystem. This patch kills the race by making permission check and the rest of open atomic w.r.t. remounting. Other file operations also have this race condition and should be fixed in similar manner.
04_brsem_convert-cpucontrol-to-brsem.patch
: convert cpucontrol to brsem
cpucontrol synchronizes cpu hotplugging and used to be a semaphore. This patch converts it to brsem.
[ End of patch descriptions ]
Thanks.
--
tejun
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
01_brsem_implement_brsem.patch
This patch implements big reader semaphore - a rwsem with very cheap reader-side operations and very expensive writer-side operations. For details, please read comments at the top of kern/brsem.c.
include/linux/brsem.h | 202 +++++++++++ init/main.c | 3 kernel/Makefile | 2 kernel/brsem.c | 869 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 1075 insertions(+), 1 deletion(-)
include/linux/brsem.h new file +202 -0
@@ -0,0 +1,202 @@ +#ifndef __LINUX_BRSEM_H +#define __LINUX_BRSEM_H + +/* +* include/linux/brsem.h - Big reader rw semaphore +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public License as +* published by the Free Software Foundation; either version 2 of the +* License, or (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, but +* WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA +* 02111-1307, USA. +* +* Copyright (C) 2005 Tejun Heo <[email protected]> +* +* See kernel/brsem.c for more information. +*/ + +#include <linux/spinlock.h> +#include <linux/interrupt.h> +#include <linux/wait.h> +#include <linux/workqueue.h> +#include <linux/percpu.h> +#include <asm/semaphore.h> + +DECLARE_PER_CPU(int *, brsem_rcnt_ar); + +struct brsem { +int idx; +spinlock_t lock; +long long master_rcnt; +wait_queue_head_t read_wait; +wait_queue_head_t write_wait; +struct semaphore write_mutex; +struct work_struct async_work; +unsigned flags; +}; + +enum { +/* +* sem->flags, protected by sem->lock +*/ +BRSEM_F_ALLOCATED =0x0001 , +BRSEM_F_BYPASS =0x0002 , + +/* Async todo flags */ +BRSEM_F_ASYNC_UPWRITE =0x0100 , +BRSEM_F_ASYNC_DESTROY =0x0200 , +BRSEM_F_ASYNC_MASK =0x0300 , +}; + +/* +* brsem subsys initialization routines, called from init/main.c +*/ +void brsem_init_early (void ); +void brsem_init (void ); + +/* +* The following initializer and __create_brsem() are for cases where +* brsem should be used before brsem_init_early() is finished. +*/ +#define BRSEM_BYPASS_INITIALIZER { .idx = 0, .flags = BRSEM_F_BYPASS } + +struct brsem *__create_brsem (struct brsem *sem); + +int __brsem_down_read_slow (struct brsem *sem,int interruptible); +int __brsem_down_read_trylock_slow (struct brsem *sem); +void __brsem_up_read_slow (struct brsem *sem); + +static inline int *__brsem_rcnt (struct brsem *sem) +{ +return __get_cpu_var (brsem_rcnt_ar) + sem->idx; +} + +/* +* The following *_crit functions can be changed to use +* local_irq_disable/enable on architectures where irq switching is +* cheaper than bh switching. See brsem.c for more information. +*/ +static inline void __brsem_enter_crit (void ) +{ +local_bh_disable (); +} + +static inline void __brsem_leave_crit (void ) +{ +if (!irqs_disabled ())/* is this cheap on all archs? */ +local_bh_enable (); +else +__local_bh_enable (); +} + +static inline int __brsem_down_read (struct brsem *sem,int interruptible) +{ +int *p; +might_sleep (); +__brsem_enter_crit (); + p =__brsem_rcnt (sem); +if (*p -1 <INT_MAX -1 ) {/* *p != INT_MIN && *p != INT_MAX */ + (*p)++; +__brsem_leave_crit (); +return 0 ; + } +return __brsem_down_read_slow (sem, interruptible); +} + +/* +* Public functions +*/ + +/** +* create_brsem - allocates and initializes a new brsem. Returns +* pointer to the new brsem on success, NULL on failure +* +* This function may sleep. +*/ +static inline struct brsem *create_brsem (void ) +{ +return __create_brsem (NULL ); +} + +void destroy_brsem (struct brsem *brsem); + +/** +* brsem_down_read - read lock the specified brsem +* +* This function may sleep. +*/ +static inline void brsem_down_read (struct brsem *sem) +{ +__brsem_down_read (sem,0 ); +} + +/** +* brsem_down_read_interruptible - read lock the specified brsem +* (interruptible). Returns 0 on success and -EINTR if interrupted. +* +* This function is identical to brsem_down_read except that it may be +* interrupted by signal. This function may sleep. +*/ +static inline int brsem_down_read_interruptible (struct brsem *sem) +{ +return __brsem_down_read (sem,1 ); +} + +/** +* brsem_down_read_trylock - try to read lock the specified brsem. +* Returns 1 on sucess and 0 on failure. +* +* If the specfied brsem can be acquired without sleeping, it's +* acquired and 1 is returned; otherwise, 0 is returned. This +* function doesn't sleep and can be called from anywhere except for +* irq handlers. +*/ +static inline int brsem_down_read_trylock (struct brsem *sem) +{ +int *p; +BUG_ON (in_irq ()); +__brsem_enter_crit (); + p =__brsem_rcnt (sem); +if (*p -1 <INT_MAX -1 ) {/* *p != INT_MIN && *p != INT_MAX */ + (*p)++; +__brsem_leave_crit (); +return 1 ; + } +return __brsem_down_read_trylock_slow (sem); +} + +/** +* brsem_up_read - read unlock the specified brsem +* +* This function doesn't sleep and can be called from anywhere except +* for irq handlers. +*/ +static inline void brsem_up_read (struct brsem *sem) +{ +int *p; +BUG_ON (in_irq ()); +__brsem_enter_crit (); + p =__brsem_rcnt (sem); +if (*p >INT_MIN +1 ) {/* *p != INT_MIN && *p != INT_MIN + 1 */ + (*p)--; +__brsem_leave_crit (); +return ; + } +__brsem_up_read_slow (sem); +} + +void brsem_down_write (struct brsem *sem); +int brsem_down_write_interruptible (struct brsem *sem); +void brsem_up_write (struct brsem *sem); +void brsem_up_write_async (struct brsem *sem); + +#endif /* __LINUX_BRSEM_H */
kernel/Makefile +1 -1
@@ -7,7 +7,7 @@ obj-y = sched.o fork.o exec_domain.o sysctl.o capability.o ptrace.o timer.o user.o \ signal.o sys.o kmod.o workqueue.o pid.o \ rcupdate.o intermodule.o extable.o params.o posix-timers.o \ - kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o + kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o brsem.o obj-$(CONFIG_FUTEX) += futex.o obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o
kernel/brsem.c new file +869 -0
869 lines not on this page · raw message
init/main.c +3 -0
@@ -35,6 +35,7 @@#include <linux/kernel_stat.h> #include <linux/security.h> #include <linux/workqueue.h> +#include <linux/brsem.h> #include <linux/profile.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h>
@@ -511,6 +512,7 @@ asmlinkage void __init start_kernel(voidkmem_cache_init ();setup_per_cpu_pageset ();numa_policy_init (); +brsem_init_early ();if (late_time_init)late_time_init ();calibrate_delay ();
@@ -605,6 +607,7 @@ static void __init do_basic_setup(void) {/* drivers will send hotplug events */ init_workqueues (); +brsem_init ();usermodehelper_init ();driver_init ();
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Quoted 7 lines · Tejun Heo
01_brsem_implement_brsem.patch
This patch implements big reader semaphore - a rwsem with very cheap reader-side operations and very expensive writer-side operations. For details, please read comments at the top of
kern/brsem.c.
This thing looks pretty overengineered. It is difficult to read with all the little wrapper functions and weird naming schemes.
What would be wrong with an array of NR_CPUS rwsems? The only tiny trick you would have to do AFAIKS is have up_read remember what rwsem down_read took, but that could be returned from down_read as a token.
I have been meaning to do something like this for mmap_sem to see what happens to page fault scalability (though the heavy write-side would make such a scheme unsuitable for mainline).
Signature · 3 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Quoted 4 lines · Nick Piggin
What would be wrong with an array of
NR_CPUSrwsems? The only tiny trick you would have to do AFAIKS is haveup_readremember what rwsemdown_readtook, but that could be returned fromdown_readas a token.
Here's something to start with. Add initialisers / cacheline alignment / percpu_alloc to taste.
For bonus points I guess you could change the implementation to just a single waitqueue mechanism with NR_CPUs counters.
In fact, it probably needs quite a bit of work before it is mainline-worthy, but as a proof of concept - do you see anything wrong with it?
Nick
Signature · 1 line
SUSE Labs, Novell Inc.
include/linux/brsem.h new file +18 -0
@@ -0,0 +1,18 @@ +#ifndef __BRSEM_H +#define __BRSEM_H + +#include <linux/rwsem.h> +struct brsem { +struct rw_semaphore cpu_sem[NR_CPUS ]; +}; + +#define BRSEM_READ_TRYLOCK_FAILED -1 +typedef int brsem_read_t ; + +brsem_read_t brsem_down_read (struct brsem *); +brsem_read_t brsem_down_read_trylock (struct brsem *); +void brsem_up_read (struct brsem *,brsem_read_t ); +void brsem_down_write (struct brsem *); +void brsem_up_write (struct brsem *); + +#endif
lib/Makefile +1 -1
@@ -5,7 +5,7 @@ lib-y := errno.o ctype.o string.o vsprintf.o cmdline.o \ bust_spinlocks.o rbtree.o radix-tree.o dump_stack.o \ idr.o div64.o int_sqrt.o bitmap.o extable.o prio_tree.o \ - sha1.o + sha1.o brsem.o lib-y += kobject.o kref.o kobject_uevent.o klist.o
lib/brsem.c new file +38 -0
@@ -0,0 +1,38 @@ +#include <linux/brsem.h> +#include <linux/rwsem.h> +#include <linux/smp.h> + +brsem_read_t brsem_down_read (struct brsem *brsem) +{ +brsem_read_t ret =smp_processor_id (); +down_read (&brsem->cpu_sem[ret]); +return ret; +} + +brsem_read_t brsem_down_read_trylock (struct brsem *brsem) +{ +brsem_read_t ret =smp_processor_id (); +if (!down_read_trylock (&brsem->cpu_sem[ret])) +return BRSEM_READ_TRYLOCK_FAILED ; +return ret; +} + +void brsem_up_read (struct brsem *brsem,brsem_read_t token) +{ +up_read (&brsem->cpu_sem[token]); +} + +void brsem_down_write (struct brsem *brsem) +{ +int i; +for (i =0 ; i <NR_CPUS ; i++) +down_write (&brsem->cpu_sem[i]); +} + +void brsem_up_write (struct brsem *brsem) +{ +int i; +for (i =0 ; i <NR_CPUS ; i++) +up_write (&brsem->cpu_sem[i]); +} +
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
In reply to Nick PigginHello, Nick.
Quoted 10 lines · Nick Piggin
Quoted 7 lines · Tejun Heo
01_brsem_implement_brsem.patch
This patch implements big reader semaphore - a rwsem with very cheap reader-side operations and very expensive writer-side operations. For details, please read comments at the top of
kern/brsem.c.
Quoted 3 lines
This thing looks pretty overengineered. It is difficult to read with all the little wrapper functions and weird naming schemes.
As I've said in the other reply, I'll first rip off three stage init
thing for cpucontrol. That added pretty much complexity to it. And with the weird naming scheme, please tell me how to fix it. I have no problem renaming things.
Quoted 4 lines
What would be wrong with an array of
NR_CPUSrwsems? The only tiny trick you would have to do AFAIKS is haveup_readremember what rwsemdown_readtook, but that could be returned fromdown_readas a token.
Using array of rwsems means that every reader-side ops performs
(unnecessary) real down and up operations on the semaphore which involve atomic memory op and probably memory barrier. These are pretty expensive operations.
What brsem tries to do is implementing rwsem semantics while
performing only normal (as opposed to atomic/barrier) intstructions during reader-side operations. That's why all the call_on_all_cpus stuff is needed while write-locking. Do you think avoiding atomic/barrier stuff would be an overkill?
Quoted 3 lines
I have been meaning to do something like this for
mmap_semto see what happens to page fault scalability (though the heavy write-side would make such a scheme unsuitable for mainline).
I agree that brsem write-locking would be too heavy for mmap_sep for
usual cases.
Thanks.
Signature · 1 line
tejun
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Quoted 4 lines · Tejun Heo
As I've said in the other reply, I'll first rip off three stage init
thing for cpucontrol. That added pretty much complexity to it. And with the weird naming scheme, please tell me how to fix it. I have no problem renaming things.
OK, my criticism of your naming was not constructive so I apologise for that. I willll help you with some of those minor issues if we establish that your overall design is a a goer.
Quoted 6 lines
Quoted 4 lines
What would be wrong with an array of
NR_CPUSrwsems? The only tiny trick you would have to do AFAIKS is haveup_readremember what rwsemdown_readtook, but that could be returned fromdown_readas a token.
Quoted 6 lines
Using array of rwsems means that every reader-side ops performs
(unnecessary) real down and up operations on the semaphore which involve atomic memory op and probably memory barrier. These are pretty expensive operations.What brsem tries to do is implementing rwsem semantics while performing
only normal (as opposed to atomic/barrier) intstructions during reader-side operations. That's why all thecall_on_all_cpusstuff is needed while write-locking. Do you think avoiding atomic/barrier stuff would be an overkill?
Yes I think so. I think the main problem on modern CPUs is not atomic operations as such, but cacheline bouncing.
Without any numbers, I'd guess that your down_read is more expensive than mine simply due to touching more cachelines and having more branches.
The other thing is simply that you really want your synchronization primitives to be as simple and verifiable as possible. For example rwsems even recently have had subtle memory ordering and other implemntation corner cases, and they're much less complex than this brsem.
Nick
Signature · 3 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Hello, Nick.
Quoted 11 lines · Nick Piggin
Quoted 4 lines · Tejun Heo
As I've said in the other reply, I'll first rip off three stage init
thing for cpucontrol. That added pretty much complexity to it. And with the weird naming scheme, please tell me how to fix it. I have no problem renaming things.OK, my criticism of your naming was not constructive so I apologise for that. I willll help you with some of those minor issues if we establish that your overall design is a a goer.
Thanks. :-)
Quoted 15 lines
Quoted 8 lines
Quoted 5 lines
What would be wrong with an array of
NR_CPUSrwsems? The only tiny trick you would have to do AFAIKS is haveup_readremember what rwsemdown_readtook, but that could be returned fromdown_readas a token.Quoted 6 lines
Using array of rwsems means that every reader-side ops performs
(unnecessary) real down and up operations on the semaphore which involve atomic memory op and probably memory barrier. These are pretty expensive operations.What brsem tries to do is implementing rwsem semantics while
performing only normal (as opposed to atomic/barrier) intstructions during reader-side operations. That's why all thecall_on_all_cpusstuff is needed while write-locking. Do you think avoiding atomic/barrier stuff would be an overkill?
Quoted 6 lines
Yes I think so. I think the main problem on modern CPUs is not atomic operations as such, but cacheline bouncing.
Without any numbers, I'd guess that your
down_readis more expensive than mine simply due to touching more cachelines and having more branches.
Other than local_bh_disable/enable(), all brsem read ops do are
1. accessing sem->idx
2. calculate per-cpu rcnt address from sem->idx
3. do one branch on the value of per-cpu rcnt
4. inc/dec per-cpu rcnt
So, it does access one more cachline and, yeap, there is one branch
for bh enabling and several more inside local_bh_enable. I'll try to get some benchmark numbers for comparison.
I'm thinking about adding down_read(&xxx->s_umount) to write(2) and
compare normal rwsem and brsem performance by repeitively writing short data into a file on a UP machine. Do you have better ideas?
Quoted 9 lines
The other thing is simply that you really want your synchronization primitives to be as simple and verifiable as possible. For example rwsems even recently have had subtle memory ordering and other implemntation corner cases, and they're much less complex than this brsem.
Nick
Thanks.
Signature · 1 line
tejun
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Quoted 10 lines · Tejun Heo
Other than local_bh_disable/enable(), all brsem read ops do are
1. accessing
sem->idx
2. calculate per-cpu rcnt address fromsem->idx
3. do one branch on the value of per-cpu rcnt
4. inc/dec per-cpu rcntSo, it does access one more cachline and, yeap, there is one branch for
bh enabling and several more insidelocal_bh_enable. I'll try to get some benchmark numbers for comparison.
Well local_bh_disable touches the preempt count too, although we can probably assume that's hot in cache.
You might also find yours has a bigger icache footprint as well.
Quoted 3 lines
I'm thinking about adding
down_read(&xxx->s_umount)to write(2) and
compare normal rwsem and brsem performance by repeitively writing short data into a file on a UP machine. Do you have better ideas?
To be honest I'd say that you wouldn't be able to measure it if you're going through a regular system call path, although such a measurement certainly won't hurt.
I don't have a better idea though. I don't think a busy loop microbenchmark is going to be very informative either, it might actually give a measurable difference but that difference probably won't be too representitive of real use :\
Signature · 3 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Hello, Nick.
Quoted 22 lines · Nick Piggin
Quoted 10 lines · Tejun Heo
Other than local_bh_disable/enable(), all brsem read ops do are
1. accessing
sem->idx
2. calculate per-cpu rcnt address fromsem->idx
3. do one branch on the value of per-cpu rcnt
4. inc/dec per-cpu rcntSo, it does access one more cachline and, yeap, there is one branch
for bh enabling and several more insidelocal_bh_enable. I'll try to get some benchmark numbers for comparison.Well
local_bh_disabletouches the preempt count too, although we can probably assume that's hot in cache.You might also find yours has a bigger icache footprint as well.
Quoted 3 lines
I'm thinking about adding
down_read(&xxx->s_umount)to write(2) and
compare normal rwsem and brsem performance by repeitively writing short data into a file on a UP machine. Do you have better ideas?
Quoted 9 lines
To be honest I'd say that you wouldn't be able to measure it if you're going through a regular system call path, although such a measurement certainly won't hurt.
I don't have a better idea though. I don't think a busy loop microbenchmark is going to be very informative either, it might actually give a measurable difference but that difference probably won't be too representitive of real use :\
I did a busy-loop microbenchmark, and I think it's informative enough. :-)
The following command is run on three versions - vanilla version, one
with read_down/up(->s_umount) added to vfs_write(), and one with brsem_read_down/up(->s_umount) added to vfs_write().
# time -p dd if=/dev/zero of=out bs=32 count=10M
The test is run three times and the results are averaged.
a. vanilla
real 58.63
user 5.61
sys 52.37
b. rwsem
real 59.24
user 6.06
sys 52.29
c. brsem
real 61.74
user 5.78
sys 55.04
I don't think brsem has any chance of being faster than rwsem if it's
slower in this micro benchmark. One weird thing is that the result of rwsem consistently showed higher user time and lower system time than vanilla (no synchronization) case, maybe oprofiling will tell something.
Anyways, you were absolutely right. My brsem was a pretty stupid idea
after all. Let's hope at least I learned something from it. :-(
Thanks a lot.
Signature · 1 line
tejun
Re: [PATCH linux-2.6 01/04] brsem: implement big reader semaphore
Quoted 2 lines · Tejun Heo
I did a busy-loop microbenchmark, and I think it's informative enough. :-)
Great!
Quoted 19 lines
The following command is run on three versions - vanilla version, one
with read_down/up(->s_umount) added tovfs_write(), and one with brsem_read_down/up(->s_umount) added tovfs_write().# time -p dd if=/dev/zero of=out bs=32 count=10MThe test is run three times and the results are averaged.
a. vanilla
real 58.63
user 5.61
sys 52.37b. rwsem
real 59.24
user 6.06
sys 52.29
Quoted 9 lines
c. brsem
real 61.74
user 5.78
sys 55.04I don't think brsem has any chance of being faster than rwsem if it's
slower in this micro benchmark. One weird thing is that the result of rwsem consistently showed higher user time and lower system time than vanilla (no synchronization) case, maybe oprofiling will tell something.
Yep, probably just some timing or cache anomaly, generally just sum the user and system time in the case where you are running identical userspace code... I wouldn't worry too much about it.
Quoted 3 lines
Anyways, you were absolutely right. My brsem was a pretty stupid idea
after all. Let's hope at least I learned something from it. :-(
I wouldn't say stupid. Implementation was too complex, but some clever ideas were required to solve the problems you identified.
There are definitely a couple of possible places where a brsem may be useful, and I think cpucontrol semaphore is one of those.
Al can probably comment on its use in the superblock. It seems fair enough, though I think we'll want to slim down my naive rwsem array and maybe think about using alloc_percpu.
Is the remount case much rarer than mount/unmount? Do we need to be careful about bloating the size of the superblock on large machines? In that case maybe a global remount brsem will be enough to handle this race?
Signature · 3 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
In reply to Tejun Heo04_brsem_convert-cpucontrol-to-brsem.patch
cpucontrol synchronizes cpu hotplugging and used to be a semaphore. This patch converts it to brsem.
include/linux/brsem.h | 4 +- include/linux/cpu.h | 16 ++++++--- init/main.c | 1 kernel/brsem.c | 38 +++++++++++++++-------- kernel/cpu.c | 81 +++++++++++++++++++++++++++++++++++++++++++------- 5 files changed, 110 insertions(+), 30 deletions(-)
include/linux/cpu.h +10 -6
@@ -23,7 +23,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> -#include <asm/semaphore.h> +#include <linux/brsem.h> struct cpu {int node_id;/* The node which contains the CPU */
@@ -59,10 +59,11 @@ extern struct sysdev_class cpu_sysdev_cl#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ -extern struct semaphore cpucontrol; -#define lock_cpu_hotplug () down(&cpucontrol) -#define unlock_cpu_hotplug () up(&cpucontrol) -#define lock_cpu_hotplug_interruptible () down_interruptible(&cpucontrol) +extern struct brsem cpucontrol; +extern struct task_struct *cpucontrol_holder; +#define lock_cpu_hotplug () brsem_down_read(&cpucontrol) +#define unlock_cpu_hotplug () brsem_up_read(&cpucontrol) +#define is_cpu_hotplug_holder () (cpucontrol_holder == current)#define hotcpu_notifier (fn, pri) { \ static struct notifier_block fn##_nb = \ { .notifier_call = fn, .priority = pri }; \
@@ -74,11 +75,14 @@ extern int __attribute__((weak)) smp_pre#else #define lock_cpu_hotplug () do { } while (0)#define unlock_cpu_hotplug () do { } while (0) -#define lock_cpu_hotplug_interruptible () 0 +#define is_cpu_hotplug_holder () 0#define hotcpu_notifier (fn, pri)/* CPUs don't go offline once they're online w/o CONFIG_HOTPLUG_CPU */ static inline int cpu_is_offline (int cpu) {return 0 ; }#endif +/* cpucontrol initializer, called from init/main.c */ +void cpucontrol_init (void ); +#endif /* _LINUX_CPU_H_ */
init/main.c +1 -0
@@ -513,6 +513,7 @@ asmlinkage void __init start_kernel(voidsetup_per_cpu_pageset ();numa_policy_init ();brsem_init_early (); +cpucontrol_init ();if (late_time_init)late_time_init ();calibrate_delay ();
kernel/cpu.c +71 -10
128 lines not on this page · raw message
kernel/brsem.c +25 -13
@@ -156,7 +156,7 @@ static void coac_schedule_work_per_cpu(vschedule_work (works +smp_processor_id ()); } -static void call_on_all_cpus (void (*func)(void *),void *data) +static void call_on_all_cpus (void (*func)(void *),void *data,int skip_cpulock) {static DECLARE_MUTEX (coac_mutex);/* serializes uses of coac_works */ static struct work_struct coac_works[NR_CPUS ];
@@ -174,7 +174,8 @@ static void call_on_all_cpus(void (*func }down (&coac_mutex); -lock_cpu_hotplug (); +if (!skip_cpulock) +lock_cpu_hotplug ();/* * If we're on keventd, scheduling work and waiting for it
@@ -202,7 +203,8 @@ static void call_on_all_cpus(void (*funcwait_for_completion (&coac_arg.completion); } -unlock_cpu_hotplug (); +if (!skip_cpulock) +unlock_cpu_hotplug ();up (&coac_mutex); }
@@ -305,7 +307,7 @@ static int expand_brsem(int target_idx) ebarg.new_len = new_len;atomic_set (&ebarg.failed,0 ); -call_on_all_cpus (expand_brsem_cpucb, &ebarg); +call_on_all_cpus (expand_brsem_cpucb, &ebarg,0 ); res = -ENOMEM ;if (atomic_read (&ebarg.failed))
@@ -405,13 +407,13 @@ static void sync_brsem_cpucb(void *data)__brsem_leave_crit (); } -static void sync_brsem (struct brsem *sem,int write_locking) +static void sync_brsem (struct brsem *sem,int write_locking,int skip_cpulock) {int cpu;if (brsem_initialized) {struct sync_brsem_arg sbarg = { sem, write_locking }; -call_on_all_cpus (sync_brsem_cpucb, &sbarg); +call_on_all_cpus (sync_brsem_cpucb, &sbarg, skip_cpulock);return ; }
@@ -420,7 +422,8 @@ static void sync_brsem(struct brsem *sem* single threaded. Sync manually. Note that lockings are * not necessary here. They're done just for consistency. */ -lock_cpu_hotplug (); +if (!skip_cpulock) +lock_cpu_hotplug ();spin_lock_crit (&sem->lock);for_each_online_cpu (cpu) {int *p =per_cpu (brsem_rcnt_ar, cpu) + sem->idx;
@@ -429,14 +432,15 @@ static void sync_brsem(struct brsem *sem *p = write_locking ?INT_MIN :0 ; }spin_unlock_crit (&sem->lock); -unlock_cpu_hotplug (); +if (!skip_cpulock) +unlock_cpu_hotplug (); }static void do_destroy_brsem (struct brsem *sem) {check_idx (sem); -sync_brsem (sem,0 ); +sync_brsem (sem,0 ,0 );BUG_ON (sem->master_rcnt !=0 );spin_lock (&brsem_alloc_lock);
@@ -586,8 +590,14 @@ void __brsem_up_read_slow(struct brsem *__brsem_leave_crit (); } +/* +* skip_cpulock is used to avoid deadlocks when write operations are +* performed on cpu hotplug brsem. See kernel/cpu.c for more +* information. +*/ static int __brsem_down_write (struct brsem *sem,int interruptible) { +int skip_cpulock =is_cpu_hotplug_holder ();int res;if (is_bypass (sem))
@@ -601,7 +611,7 @@ static int __brsem_down_write(struct brs }else down (&sem->write_mutex); -sync_brsem (sem,1 ); +sync_brsem (sem,1 , skip_cpulock);spin_lock_crit (&sem->lock);
@@ -619,7 +629,7 @@ static int __brsem_down_write(struct brsfinish_wait (&sem->write_wait, &wait);if (interruptible &&signal_pending (current)) { -sync_brsem (sem,0 ); +sync_brsem (sem,0 , skip_cpulock);wake_up_all (&sem->read_wait);up (&sem->write_mutex);return -EINTR ;
@@ -661,12 +671,14 @@ int brsem_down_write_interruptible(struc*/ void brsem_up_write (struct brsem *sem) { +int skip_cpulock =is_cpu_hotplug_holder (); +if (is_bypass (sem))return ;check_idx (sem);BUG_ON (sem->master_rcnt); -sync_brsem (sem,0 ); +sync_brsem (sem,0 , skip_cpulock);wake_up_all (&sem->read_wait);up (&sem->write_mutex); }
@@ -856,7 +868,7 @@ void __init brsem_init(void) brsem_initialized =1 ;/* Make sure other cpus see above change */ -call_on_all_cpus (dummy_cpucb,NULL ); +call_on_all_cpus (dummy_cpucb,NULL ,0 ); }EXPORT_SYMBOL (__create_brsem);
include/linux/brsem.h +3 -1
@@ -65,7 +65,9 @@ void brsem_init(void);/* * The following initializer and __create_brsem() are for cases where -* brsem should be used before brsem_init_early() is finished. +* brsem should be used before brsem_init_early() is finished. If +* you're trying to use brsem for such cases, refer to +* include/linux/cpu.h and kernel/cpu.c for example. */ #define BRSEM_BYPASS_INITIALIZER { .idx = 0, .flags = BRSEM_F_BYPASS }
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
Quoted 22 lines · Tejun Heo
+/*
+ * cpucontrol is a brsem used to synchronize cpu hotplug events.
+ * Invokinglock_cpu_hotplug()read-locks cpucontrol and no
+ * hotplugging events will occur until it's released.
+ *
+ * Unfortunately, brsem itself makes use oflock_cpu_hotplug()and
+ * performing brsem write-lock operations on cpucontrol deadlocks.
+ * This is avoided by...
+ *
+ * a. guaranteeing that cpu hotplug events won't occur during the
+ * write-lock operations, and
+ *
+ * b. skippinglock_cpu_hotplug()inside brsem.
+ *
+ * #a is achieved by acquiring and releasingcpucontrol_mutexoutside
+ * cpucontrol write-lock. #b is achieved by skipping
+ *lock_cpu_hotplug()inside brsem if the current task is
+ *cpucontrol_mutexholder (is_cpu_hotplug_holder()test).
+ *
+ * Also, note that cpucontrol is first initialized with
+ *BRSEM_BYPASS_INITIALIZERand then initialized again with
+ *__create_brsem()instead of simply usingcreate_brsem(). This is
Quoted 8 lines
+ * necessary as cpucontrol brsem gets used way before brsem subsystem
+ * becomes up and running.
+ *
+ * Until brsem is properly initialized, all brsem ops succeed
+ * unconditionally. cpucontrol becomes operational only after
+ *cpucontrol_init()is finished, which should be called after
+ *brsem_init_early().
+ */
Mmm, this is just insane IMO.
Note that I happen to also think the idea (brsems) have merit, and that cpucontrol may be one of the places where a sane implementation would actually be useful... but at least when you're introducing this kind of complexity anywhere, you *really* need to be able to back it up with numbers.
As far as the VFS race fix goes, I guess Al or someone else will comment on its correctness. But I think it might be nicer to first fix it with a regular rwsem and then show some numbers to justify its conversion to a brsem.
If you need interruptible rwsems, I almost got an implementation working a while back, and David Howells recently said he was interested in doing them... so that's not an impossibility.
Nick
Signature · 3 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
Quoted 47 lines · Nick Piggin
Quoted 22 lines · Tejun Heo
+/*
+ * cpucontrol is a brsem used to synchronize cpu hotplug events.
+ * Invokinglock_cpu_hotplug()read-locks cpucontrol and no
+ * hotplugging events will occur until it's released.
+ *
+ * Unfortunately, brsem itself makes use oflock_cpu_hotplug()and
+ * performing brsem write-lock operations on cpucontrol deadlocks.
+ * This is avoided by...
+ *
+ * a. guaranteeing that cpu hotplug events won't occur during the
+ * write-lock operations, and
+ *
+ * b. skippinglock_cpu_hotplug()inside brsem.
+ *
+ * #a is achieved by acquiring and releasingcpucontrol_mutexoutside
+ * cpucontrol write-lock. #b is achieved by skipping
+ *lock_cpu_hotplug()inside brsem if the current task is
+ *cpucontrol_mutexholder (is_cpu_hotplug_holder()test).
+ *
+ * Also, note that cpucontrol is first initialized with
+ *BRSEM_BYPASS_INITIALIZERand then initialized again with
+ *__create_brsem()instead of simply usingcreate_brsem(). This isQuoted 8 lines
+ * necessary as cpucontrol brsem gets used way before brsem subsystem
+ * becomes up and running.
+ *
+ * Until brsem is properly initialized, all brsem ops succeed
+ * unconditionally. cpucontrol becomes operational only after
+ *cpucontrol_init()is finished, which should be called after
+ *brsem_init_early().
+ */Mmm, this is just insane IMO.
Note that I happen to also think the idea (brsems) have merit, and that cpucontrol may be one of the places where a sane implementation would actually be useful... but at least when you're introducing this kind of complexity anywhere, you *really* need to be able to back it up with numbers.
As far as the VFS race fix goes, I guess Al or someone else will comment on its correctness. But I think it might be nicer to first fix it with a regular rwsem and then show some numbers to justify its conversion to a brsem.
Quoted 6 lines
If you need interruptible rwsems, I almost got an implementation working a while back, and David Howells recently said he was interested in doing them... so that's not an impossibility.
Nick
Hello, Nick.
I do agree that it's absolutely ugly. I thought about ripping the
3-tage init'ing and cpu hotplug stuff as currently cpu hotplug locking isn't used frequently, but was just giving a shot. I'll strip this thing out.
Thanks.
Signature · 1 line
tejun
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
In reply to Nick PigginQuoted 33 lines · Nick Piggin
Quoted 22 lines · Tejun Heo
+/*
+ * cpucontrol is a brsem used to synchronize cpu hotplug events.
+ * Invokinglock_cpu_hotplug()read-locks cpucontrol and no
+ * hotplugging events will occur until it's released.
+ *
+ * Unfortunately, brsem itself makes use oflock_cpu_hotplug()and
+ * performing brsem write-lock operations on cpucontrol deadlocks.
+ * This is avoided by...
+ *
+ * a. guaranteeing that cpu hotplug events won't occur during the
+ * write-lock operations, and
+ *
+ * b. skippinglock_cpu_hotplug()inside brsem.
+ *
+ * #a is achieved by acquiring and releasingcpucontrol_mutexoutside
+ * cpucontrol write-lock. #b is achieved by skipping
+ *lock_cpu_hotplug()inside brsem if the current task is
+ *cpucontrol_mutexholder (is_cpu_hotplug_holder()test).
+ *
+ * Also, note that cpucontrol is first initialized with
+ *BRSEM_BYPASS_INITIALIZERand then initialized again with
+ *__create_brsem()instead of simply usingcreate_brsem(). This isQuoted 8 lines
+ * necessary as cpucontrol brsem gets used way before brsem subsystem
+ * becomes up and running.
+ *
+ * Until brsem is properly initialized, all brsem ops succeed
+ * unconditionally. cpucontrol becomes operational only after
+ *cpucontrol_init()is finished, which should be called after
+ *brsem_init_early().
+ */
Quoted 7 lines
Mmm, this is just insane IMO.
Note that I happen to also think the idea (brsems) have merit, and that cpucontrol may be one of the places where a sane implementation would actually be useful... but at least when you're introducing this kind of complexity anywhere, you *really* need to be able to back it up with numbers.
The only performance-related complaint with cpu hotplug of which I'm aware -- that taking a cpu down on a large system can be painfully slow -- resides in the "write side" of the code, which is not the case that the brsem implementation optimizes. I think this patch would make that case even worse. So I don't think it's appropriate to use a brsem for cpu hotplug, especially without trying rwsem first.
Nathan
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
Quoted 10 lines · Nathan Lynch
Quoted 7 lines · Nick Piggin
Note that I happen to also think the idea (brsems) have merit, and that cpucontrol may be one of the places where a sane implementation would actually be useful... but at least when you're introducing this kind of complexity anywhere, you *really* need to be able to back it up with numbers.
Quoted 8 lines
The only performance-related complaint with cpu hotplug of which I'm aware -- that taking a cpu down on a large system can be painfully slow -- resides in the "write side" of the code, which is not the case that the brsem implementation optimizes. I think this patch would make that case even worse. So I don't think it's appropriate to use a brsem for cpu hotplug, especially without trying rwsem first.
I'm not sure that a brsem would make a noticable difference.
It isn't that cpu hotplug semaphore is a performance problem now, but that it isn't being used in as many cases as it could be due to its unscalable nature. For example, a while back I wanted to use it in the fork() path in the scheduler but couldn't.
Anyway, as I said, you need to be able to back it up with numbers ;)
Nick
Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH linux-2.6 04/04] brsem: convert cpucontrol to brsem
Hello, Nathan & Nick.
Quoted 12 lines · Nick Piggin
Quoted 10 lines · Nathan Lynch
Quoted 7 lines · Nick Piggin
Note that I happen to also think the idea (brsems) have merit, and that cpucontrol may be one of the places where a sane implementation would actually be useful... but at least when you're introducing this kind of complexity anywhere, you *really* need to be able to back it up with numbers.
Quoted 8 lines
Quoted 8 lines
The only performance-related complaint with cpu hotplug of which I'm aware -- that taking a cpu down on a large system can be painfully slow -- resides in the "write side" of the code, which is not the case that the brsem implementation optimizes. I think this patch would make that case even worse. So I don't think it's appropriate to use a brsem for cpu hotplug, especially without trying rwsem first.
Actually, a patch which converts cpucontrol to rwsem was once in -mm. I don't know what happened to it. I can't see it in 2.6.13-rc2-mm1.
Quoted 8 lines
I'm not sure that a brsem would make a noticable difference.
It isn't that cpu hotplug semaphore is a performance problem now, but that it isn't being used in as many cases as it could be due to its unscalable nature. For example, a while back I wanted to use it in the
fork()path in the scheduler but couldn't.
I couldn't have put it better. As it currently stands, cpucontrol
doesn't have any heavy readers. It's partly because it's not necessary but at least for some part it's because it cannot be used in such cases as the overhead is too big. The same is true for super->s_umount rwsem - read_down'ing per-fs rwsem on every write(2) will hurt performance on SMP machines. I think we'll have more and more of this class of synchronization problems as we add hotplug capability to subsystems.
Having spent a week on implementing something very ugly :-), I'm a bit
embarrased here but I still want to point out that we need something to solve this problem.
Quoted 2 lines
Anyway, as I said, you need to be able to back it up with numbers ;)
Right, gotta benchmark before committing full implementation.
Thanks.
Signature · 1 line
tejun
Re: [PATCH linux-2.6 02/04] brsem: convert super_block->s_umount to brsem
In reply to Tejun Heo02_brsem_convert-s_umount-to-brsem.patch
Convert super_block->s_umount from rwsem to brsem.
arch/um/drivers/mconsole_kern.c | 2 - fs/9p/vfs_super.c | 2 - fs/afs/super.c | 2 - fs/cifs/cifsfs.c | 2 - fs/fs-writeback.c | 8 ++--- fs/jffs2/super.c | 2 - fs/libfs.c | 2 - fs/namespace.c | 8 ++--- fs/nfs/inode.c | 4 +- fs/quota.c | 4 +- fs/reiserfs/procfs.c | 2 - fs/super.c | 60 +++++++++++++++++++++------------------- include/linux/fs.h | 3 +- security/selinux/hooks.c | 2 - 14 files changed, 54 insertions(+), 49 deletions(-)
fs/9p/vfs_super.c +1 -1
@@ -189,7 +189,7 @@ static struct super_block *v9fs_get_sb(s put_back_sb:/* deactivate_super calls v9fs_kill_super which will frees the rest */ -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);deactivate_super (sb);return ERR_PTR (retval); }
fs/afs/super.c +1 -1
@@ -343,7 +343,7 @@ static struct super_block *afs_get_sb(st ret =afs_fill_super (sb, ¶ms, flags &MS_VERBOSE ?1 :0 );if (ret <0 ) { -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);deactivate_super (sb); goto error; }
fs/cifs/cifsfs.c +1 -1
@@ -435,7 +435,7 @@ cifs_get_sb(struct file_system_type *fs_ rc =cifs_read_super (sb, data, dev_name, flags &MS_VERBOSE ?1 :0 );if (rc) { -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);deactivate_super (sb);return ERR_PTR (rc); }
fs/fs-writeback.c +4 -4
@@ -425,13 +425,13 @@ restart:* waiting around, most of the time the FS is going to * be unmounted by the time it is released. */ -if (down_read_trylock (&sb->s_umount)) { +if (brsem_down_read_trylock (sb->s_umount)) {if (sb->s_root) {spin_lock (&inode_lock);sync_sb_inodes (sb, wbc);spin_unlock (&inode_lock); } -up_read (&sb->s_umount); +brsem_up_read (sb->s_umount); }spin_lock (&sb_lock);if (__put_super_and_need_restart (sb))
@@ -516,12 +516,12 @@ restart: sb->s_syncing =1 ; sb->s_count++;spin_unlock (&sb_lock); -down_read (&sb->s_umount); +brsem_down_read (sb->s_umount);if (sb->s_root) {sync_inodes_sb (sb, wait);sync_blockdev (sb->s_bdev); } -up_read (&sb->s_umount); +brsem_up_read (sb->s_umount);spin_lock (&sb_lock);if (__put_super_and_need_restart (sb)) goto restart;
fs/jffs2/super.c +1 -1
@@ -156,7 +156,7 @@ static struct super_block *jffs2_get_sb_if (ret) {/* Failure case... */ -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);deactivate_super (sb);return ERR_PTR (ret); }
fs/libfs.c +1 -1
@@ -233,7 +233,7 @@ get_sb_pseudo(struct file_system_type *freturn s; Enomem: -up_write (&s->s_umount); +brsem_up_write (s->s_umount);deactivate_super (s);return ERR_PTR (-ENOMEM ); }
fs/namespace.c +4 -4
@@ -421,14 +421,14 @@ static int do_umount(struct vfsmount *mn* Special case for "unmounting" root ... * we just try to remount it readonly. */ -down_write (&sb->s_umount); +brsem_down_write (sb->s_umount);if (!(sb->s_flags &MS_RDONLY )) {lock_kernel ();DQUOT_OFF (sb); retval =do_remount_sb (sb,MS_RDONLY ,NULL ,0 );unlock_kernel (); } -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);return retval; }
@@ -681,11 +681,11 @@ static int do_remount(struct nameidata *if (nd->dentry != nd->mnt->mnt_root)return -EINVAL ; -down_write (&sb->s_umount); +brsem_down_write (sb->s_umount); err =do_remount_sb (sb, flags, data,0 );if (!err) nd->mnt->mnt_flags=mnt_flags; -up_write (&sb->s_umount); +brsem_up_write (sb->s_umount);if (!err)security_sb_post_remount (nd->mnt, flags, data);return err;
fs/nfs/inode.c +2 -2
@@ -1574,7 +1574,7 @@ static struct super_block *nfs_get_sb(st error =nfs_fill_super (s, data, flags &MS_VERBOSE ?1 :0 );if (error) { -up_write (&s->s_umount); +brsem_up_write (s->s_umount);deactivate_super (s);return ERR_PTR (error); }
@@ -1931,7 +1931,7 @@ static struct super_block *nfs4_get_sb(s error =nfs4_fill_super (s, data, flags &MS_VERBOSE ?1 :0 );if (error) { -up_write (&s->s_umount); +brsem_up_write (s->s_umount);deactivate_super (s);return ERR_PTR (error); }
fs/quota.c +2 -2
@@ -209,10 +209,10 @@ restart:continue ; sb->s_count++;spin_unlock (&sb_lock); -down_read (&sb->s_umount); +brsem_down_read (sb->s_umount);if (sb->s_root && sb->s_qcop->quota_sync)quota_sync_sb (sb, type); -up_read (&sb->s_umount); +brsem_up_read (sb->s_umount);spin_lock (&sb_lock);if (__put_super_and_need_restart (sb)) goto restart;
fs/reiserfs/procfs.c +1 -1
@@ -422,7 +422,7 @@ static void *r_start(struct seq_file *m,if (IS_ERR (sget (&reiserfs_fs_type, test_sb, set_sb, s)))return NULL ; -up_write (&s->s_umount); +brsem_up_write (s->s_umount);if (de->deleted) {deactivate_super (s);
fs/super.c +32 -28
187 lines not on this page · raw message
security/selinux/hooks.c +1 -1
@@ -4407,7 +4407,7 @@ next_sb: sb->s_count++;spin_unlock (&sb_lock);spin_unlock (&sb_security_lock); -down_read (&sb->s_umount); +brsem_down_read (sb->s_umount);if (sb->s_root)superblock_doinit (sb,NULL );drop_super (sb);
include/linux/fs.h +2 -1
@@ -216,6 +216,7 @@ extern int dir_notify_enable;#include <linux/prio_tree.h> #include <linux/init.h> #include <linux/sched.h> +#include <linux/brsem.h> #include <asm/atomic.h> #include <asm/semaphore.h>
@@ -771,7 +772,7 @@ struct super_block {unsigned long s_flags;unsigned long s_magic;struct dentry *s_root; -struct rw_semaphore s_umount; +struct brsem *s_umount;struct semaphore s_lock;int s_count;int s_syncing;
arch/um/drivers/mconsole_kern.c +1 -1
@@ -150,7 +150,7 @@ void mconsole_proc(struct mc_request *remconsole_reply (req,"Failed to get procfs superblock" ,1 ,0 ); goto out; } -up_write (&super->s_umount); +brsem_up_write (super->s_umount); nd.dentry = super->s_root; nd.mnt =NULL ;
Re: [PATCH linux-2.6 03/04] brsem: fix ro-remount <-> open race condition
In reply to Tejun Heo03_brsem_fix-remount-open-race.patch
A file can be opened rw while ro-remounting is in progress resulting in open rw file on ro mounted filesystem. This patch kills the race by making permission check and the rest of open atomic w.r.t. remounting. Other file operations also have this race condition and should be fixed in similar manner.
namei.c | 11 ++++++++++- open.c | 12 ++++++++++-- 2 files changed, 20 insertions(+), 3 deletions(-)
fs/namei.c +10 -1
@@ -1412,6 +1412,12 @@ int may_open(struct nameidata *nd, int a* 11 - read/write permissions needed * which is a lot more logical, and also allows the "no perm" needed * for symlinks (where the permissions are checked later). +* +* nd->mnt->mnt_sb->s_umount brsem is read-locked on successful return +* from this function. This is to make permission checking and the +* actual open operation atomic w.r.t. remounting. The caller must +* release s_umount after open is complete. +* * SMP-safe */ int open_namei (const char * pathname,int flag,int mode,struct nameidata *nd)
@@ -1512,9 +1518,12 @@ do_last:if (path.dentry->d_inode &&S_ISDIR (path.dentry->d_inode->i_mode)) goto exit; ok: +brsem_down_read (nd->mnt->mnt_sb->s_umount); error =may_open (nd, acc_mode, flag); -if (error) +if (error) { +brsem_up_read (nd->mnt->mnt_sb->s_umount); goto exit; + }return 0 ; exit_dput:
fs/open.c +10 -2
@@ -828,8 +828,16 @@ struct file *filp_open(const char * filereturn ERR_PTR (error); error =open_namei (filename, namei_flags, mode, &nd); -if (!error) -return __dentry_open (nd.dentry, nd.mnt, flags, f); +if (!error) { + f =__dentry_open (nd.dentry, nd.mnt, flags, f); +/* +* On successful return from open_namei(), s_umount is +* read-locked, see comment above open_namei() for +* more information. +*/ +brsem_up_read (nd.mnt->mnt_sb->s_umount); +return f; + }put_filp (f);return ERR_PTR (error);