Deadlock on the mm->mmap_sem
Subsystems (21)
Quoted 4 lines
What happens is that
proc_pid_read_mapsgrabs themmap_semas a reader, and *while it holds the lock*, does acopy_to_user. This can of course page-fault, and the handler will also grab themmap_sem(if it is the same task).
Ok, that's a bug.
You must not call copy_to_user with the mmap semaphore acquired - linux semaphores are not recursive.
Quoted 2 lines
Any ideas how to fix this? Should
proc_pid_read_mapsjust drop the lock beforecopy_to_user?
Yes, and preferable switch to multiline copies - a full page temporary buffer is allocated, transfering data on a line-by-line base is way too much overhead (and the current volatile_task is an ugly hack).
--
Manfred
[ David, Andrea - can you check this out? ]
In article <001701c13fc2$cda19a90$010411ac@local>,
Quoted 8 lines · Manfred Spraul
Quoted 4 lines
What happens is that
proc_pid_read_mapsgrabs themmap_semas a reader, and *while it holds the lock*, does acopy_to_user. This can of course page-fault, and the handler will also grab themmap_sem(if it is the same task).Ok, that's a bug.
You must not callcopy_to_userwith the mmap semaphore acquired - linux semaphores are not recursive.
No, that's not the bug.
The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "copy_to_user()" and friends while holding the read lock.
The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block new readers, exactly because of this situation.
The situation wrt read-write spinlocks is exactly the same, btw, except there we have "readers can have interrupts enabled even if interrupts also take read locks" instead of having user-level faults.
Why do we want to explicitly allow this behaviour wrt mmap_sem? Because some things are inherently racy without it (ie threaded processes that read or write the address space - coredumping, ptrace etc).
Linus
Quoted 9 lines · Linus Torvalds
[ David, Andrea - can you check this out? ]
In article <001701c13fc2$cda19a90$010411ac@local>,
Quoted 5 lines · Manfred Spraul
Quoted 4 lines
What happens is that
proc_pid_read_mapsgrabs themmap_semas a reader, and *while it holds the lock*, does acopy_to_user. This can of course page-fault, and the handler will also grab themmap_sem(if it is the same task).
Quoted 5 lines
Quoted 3 lines
Ok, that's a bug.
You must not callcopy_to_userwith the mmap semaphore acquired - linux semaphores are not recursive.No, that's not the bug.
agreed.
Quoted 6 lines
The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "
copy_to_user()" and friends while holding the read lock.The bug appears to be in the implementation of the write semaphore -
down_write()doesn't undestand that blocked writes must not block new readers, exactly because of this situation.
Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to read_lock without clearing irqs). Thanks so much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first down_write (for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock). The fix is a few liner for my implementation, here it is:
lib/rwsem_spinlock.c renamed from lib/rwsem_spinlock.c.~1~ +3 -1
@@ -73,11 +73,13 @@void down_read (struct rw_semaphore *sem) { +int count;CHECK_MAGIC (sem->__magic);spin_lock (&sem->lock); + count = sem->count; sem->count +=RWSEM_READ_BIAS ; -if (__builtin_expect (sem->count,0 ) <0 ) +if (__builtin_expect (count <0 && !(count &RWSEM_READ_MASK ),0 ))rwsem_down_failed (sem,RWSEM_READ_BLOCKING_BIAS );spin_unlock (&sem->lock); }
it will be applied to next -aa. For the mainline semaphores I assume David will take care of that.
For the record, I'm using spinlock based rwsemphores. Last time I checked my asm semaphores I found a small race in up_write, I didn't checked if the mainlines semaphores were affected too but I just preferred to stay safe with the spinlock in the meantime (in the microbenchmark the spinlock based rwsems weren't that much slower [and my optimized version is much faster than the mainline spinlock based rwsem] so using asm it's not a noticeable improvement in the macro real life benchmarks and the robustness of the spinlock is quite unvaluable, even more now that allowed me to do a bugfix without panicing in doing those changes). I think I will return to the asm rwsem only after proofing my implementation with math or after writing an automated simulation that checks their correctness in all possible race combinations (assuming they're mutex and with a variable number of threads).
Andrea
From: "Andrea Arcangeli" <[email protected]>
Quoted 1 line
Quoted 1 line
The mmap semaphore is a read-write semaphore, and it _is_
permissible to
Quoted 4 lines
Quoted 4 lines
call "
copy_to_user()" and friends while holding the read lock.The bug appears to be in the implementation of the write semaphore -
down_write()doesn't undestand that blocked writes must not block
new
Quoted 4 lines
Quoted 1 line
readers, exactly because of this situation.
Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to
read_lockwithout clearing irqs). Thanks
so
Quoted 4 lines
much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first
down_write(for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).
i386 has a fair rwsemaphore, too - probably other archs must be modified as well.
Quoted 3 lines
The fix is a few liner for my
implementation, here it is:
Obivously your patch fixes the race, but we could starve down_write() if there are many page faults.
Which multithreaded apps rely on mmap for file io? innd, perhaps samba if mmap is enabled (I'm not sure what's the default and if samba is multithreaded).
If you compile a kernel for 80386, then i386 uses the generic semaphores.
Could someone with innd compile his kernel for i386, apply Andrea's patch and check that the performance doesn't break down?
IMHO modifying proc_pid_read_maps() is far simpler - I'm not aware of another recursive mmap_sem user.
--
Manfred
Quoted 12 lines · Manfred Spraul
From: "Andrea Arcangeli" <[email protected]>
Quoted 1 line
Quoted 1 line
The mmap semaphore is a read-write semaphore, and it _is_
permissible to
Quoted 4 lines
Quoted 4 lines
call "
copy_to_user()" and friends while holding the read lock.The bug appears to be in the implementation of the write semaphore -
down_write()doesn't undestand that blocked writes must not blocknew
Quoted 4 lines
Quoted 1 line
readers, exactly because of this situation.
Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to
read_lockwithout clearing irqs). Thanks
Quoted 8 lines
so
Quoted 4 lines
much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first
down_write(for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).i386 has a fair rwsemaphore, too - probably other archs must be modified as well.
yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.
Quoted 6 lines
Quoted 3 lines
The fix is a few liner for my
implementation, here it is:Obivously your patch fixes the race, but we could starve
down_write()if there are many page faults.
Yes.
Quoted 2 lines
IMHO modifying
proc_pid_read_maps()is far simpler - I'm not aware of another recursivemmap_semuser.
if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a down_read_fair that will block at if there's a down_write under us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks: read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix in proc_pid_read_maps the down_read over there could be converted to a down_read_fair (but that's just an exercise, if the page fault isn't fair it doesn't worth to have proc_pid_read_maps fair either).
Andrea
Quoted 7 lines
Quoted 3 lines
i386 has a fair rwsemaphore, too - probably other archs must be modified as well.
yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.
Wasn't there a problem with unfair rw-semaphores? I can't remember exactly now...
Quoted 15 lines
Quoted 2 lines
IMHO modifying
proc_pid_read_maps()is far simpler - I'm not aware of another recursivemmap_semuser.if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a
down_read_fairthat will block at if there's adown_writeunder us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks:read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix inproc_pid_read_mapsthedown_readover there could be converted to adown_read_fair(but that's just an exercise, if the page fault isn't fair it doesn't worth to haveproc_pid_read_mapsfair either).
If this were to be done, I'd prefer to keep down_read() as being fair and add a down_read_unfair(). This'd have the least impact on the current behaviour, and I suspect we actually want fairness most of the time.
Of course, I'd personally prefer to avoid recursive semaphore situations where possible too... it sounds far too much like trouble waiting to happen, but we can't have everything.
David
Quoted 7 lines · Linus Torvalds
The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "
copy_to_user()" and friends while holding the read lock.The bug appears to be in the implementation of the write semaphore -
down_write()doesn't undestand that blocked writes must not block new readers, exactly because of this situation.
Quoted 7 lines
The situation wrt read-write spinlocks is exactly the same, btw, except there we have "readers can have interrupts enabled even if interrupts also take read locks" instead of having user-level faults.
Why do we want to explicitly allow this behaviour wrt
mmap_sem? Because some things are inherently racy without it (ie threaded processes that read or write the address space - coredumping, ptrace etc).
Hmmm... I don't think this is possible with XADD based semaphores as they stand (my version or Andrea's).
With the current XADD based stuff, you can't distinguish between one writer running and a queue of sleeping locks and one reader running and a queue of sleeping locks without counting the sleepers
Sem(sleepers) Proc 1 Proc 2 Proc 3 Proc 4 Proc 5 ======== ====== ====== ====== ====== ====== 00000000(0) -->down_read() <--down_read() 00000001(0) -->down_write() -->down_write_failed() [schedule] FFFF0001(1) -->down_write() -->down_write_failed() [schedule] FFFE0001(2) -->down_write() -->down_write_failed() [schedule] FFFD0001(3) -->down_read_unfair() FFFC0002(3) is the active proc R or W? Sem Proc 1 Proc 2 Proc 3 Proc 4 ======== ====== ====== ====== ====== 00000000(0) -->down_write() <--down_write() FFFF0001(0) -->down_write() -->down_write_failed() [schedule] FFFE0001(1) -->down_write() -->down_write_failed() [schedule] FFFD0001(2) -->down_read_unfair() FFFC0002(2) is the active proc R or W?
In fact, it's worse than that: you can't tell the difference between two active readers and a queue of sleepers and one active writer, one failed read or write attempt as yet unprocessed, and a queue of sleepers
Sem(sleepers) Proc 1 Proc 2 Proc 3 Proc 4 Proc 5 ============== ====== ====== ====== ====== ====== 00000000(0) -->down_read() <--down_read() 00000001(0) -->down_read() <--down_read() 00000002(0) -->down_write() FFFF0003(0) -->down_write_failed() [schedule] FFFF0002(1) -->down_write() FFFE0003(1) -->down_read_unfair() FFFE0004(1) since the LSW>2 does this mean there are 2+ readers active? Sem(sleepers) Proc 1 Proc 2 Proc 3 Proc 4 Proc 5 ============== ====== ====== ====== ====== ====== 00000000(0) -->down_write() <--down_write() FFFF0001(0) -->down_write() FFFE0002(0) -->down_write_failed() [schedule] FFFE0001(1) -->down_read() FFFE0002(1) -->down_read() FFFE0003(1) -->down_read_unfair() FFFE0004(1) since the LSW>2 does this mean there are 2+ readers active?
I think it might well be too hard to do unfair reads with the XADD based stuff. The problem is that you can't compensate for the effect on the counter of failed attempts to get read or write locks, even when you've got the semaphore spinlock (the queue length is of no help).
I think that this problem can only be solved by going to the spinlock version, and maintaining a flag to say what sort of lock is currently active.
David
Quoted 1 line
Quoted 1 line
IMHO modifying
proc_pid_read_maps()is far simpler - I'm not aware
of
Quoted 5 lines
Quoted 1 line
another recursive
mmap_semuser.if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks.
But shouldn't that change wait until 2.5? Especially since another huge mm change was just merged?proc_pid_read_maps contains further bugs - afaics it it could skip lines if PAGE_SIZE > 4096 and a file path is nearly 4096 bytes long. I'll post a patch to proc_pid_read_maps - modifying the rw semaphore behaviour just asks for trouble.
--
Manfred
Quoted 27 lines · Andrea Arcangeli
Quoted 12 lines · Manfred Spraul
From: "Andrea Arcangeli" <[email protected]>
Quoted 1 line
Quoted 1 line
The mmap semaphore is a read-write semaphore, and it _is_
permissible to
Quoted 4 lines
Quoted 4 lines
call "
copy_to_user()" and friends while holding the read lock.The bug appears to be in the implementation of the write semaphore -
down_write()doesn't undestand that blocked writes must not blocknew
Quoted 4 lines
Quoted 1 line
readers, exactly because of this situation.
Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to
read_lockwithout clearing irqs). ThanksQuoted 8 lines
so
Quoted 4 lines
much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first
down_write(for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).i386 has a fair rwsemaphore, too - probably other archs must be modified as well.
yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.
Quoted 12 lines
if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a
down_read_fairthat will block at if there's adown_writeunder us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks:read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix inproc_pid_read_mapsthedown_readover there could be converted to adown_read_fair(but that's just an exercise, if the page fault isn't fair it doesn't worth to haveproc_pid_read_mapsfair either).
Be careful; If another user can grab your semaphore for read for a short time (eg for "top" or similar usage), he can construct several threads that
do this in a busy loop; the end result is that this evil user is capable of blocking out writers FOREVER if semaphores are unfair; nice DoS....
Attached is a rewritten proc_pid_read_maps. Patch against 2.4.9-ac9, but should apply against 2.4.9 as well.
As a side-effect, it's more efficient since it tries to return multiple lines in each call, without the volatile_task hack.
Code that's older than 2.4.3 can't cause recursions - that was before the rw-mmap_sem change.
* ptrace doesn't cause any recursions, it always copies into temporary kernel buffers.
* the multithreaded coredump patch added down_read() around binfmt->core_dump(). I'll think about the simplest way to fix that.
--
Manfred
fs/proc/array.c +105 -99
@@ -537,136 +537,142 @@#define MAPS_LINE_FORMAT8 "%016lx-%016lx %s %016lx %s %lu"#define MAPS_LINE_MAX8 73/* sum of 16 1 16 1 4 1 16 1 5 1 10 1 */ -#define MAPS_LINE_MAX MAPS_LINE_MAX8 +#define MAPS_LINE_FORMAT (sizeof(void*) == 4 ? MAPS_LINE_FORMAT4 : MAPS_LINE_FORMAT8) +#define MAPS_LINE_MAX (sizeof(void*) == 4 ? MAPS_LINE_MAX4 : MAPS_LINE_MAX8) +int proc_pid_maps_get_line (char *buf,struct vm_area_struct *map) +{ +/* produce the next line */ +char *line; +char str[5 ]; +int flags; +kdev_t dev; +unsigned long ino; +int len; + + flags = map->vm_flags; + + str[0 ] = flags &VM_READ ?'r' :'-' ; + str[1 ] = flags &VM_WRITE ?'w' :'-' ; + str[2 ] = flags &VM_EXEC ?'x' :'-' ; + str[3 ] = flags &VM_MAYSHARE ?'s' :'p' ; + str[4 ] =0 ; + + dev =0 ; + ino =0 ; +if (map->vm_file !=NULL ) { + dev = map->vm_file->f_dentry->d_inode->i_dev; + ino = map->vm_file->f_dentry->d_inode->i_ino; + line =d_path (map->vm_file->f_dentry, + map->vm_file->f_vfsmnt, + buf,PAGE_SIZE ); + buf[PAGE_SIZE -1 ] ='\n' ; + line -=MAPS_LINE_MAX ; +if (line < buf) + line = buf; + }else + line = buf; + + len =sprintf (line, +MAPS_LINE_FORMAT , + map->vm_start, map->vm_end, str, map->vm_pgoff <<PAGE_SHIFT , +kdevname (dev), ino); + +if (map->vm_file) { +int i; +for (i = len; i <MAPS_LINE_MAX ; i++) + line[i] =' ' ; + len = buf +PAGE_SIZE - line; +memmove (buf, line, len); + }else + line[len++] ='\n' ; +return len; +}ssize_t proc_pid_read_maps (struct task_struct *task,struct file * file,char * buf,size_t count,loff_t *ppos) {struct mm_struct *mm;struct vm_area_struct * map, * next; -char * destptr = buf, * buffer; -loff_t lineno; -ssize_t column, i; -int volatile_task; +char *tmp, *kbuf;long retval; +int off, lineno, loff; +/* reject calls with out of range parameters immediately */ + retval =0 ; +if (*ppos >LONG_MAX ) + goto out; +if (count ==0 ) + goto out; + off = (long )*ppos;/* * We might sleep getting the page, so get it first. */ retval = -ENOMEM ; - buffer = (char *)__get_free_page (GFP_KERNEL ); -if (!buffer) + kbuf = (char *)__get_free_page (GFP_KERNEL ); +if (!kbuf) goto out; -if (count ==0 ) - goto getlen_out; + tmp = (char *)__get_free_page (GFP_KERNEL ); +if (!tmp) + goto out_free1; +task_lock (task); mm = task->mm;if (mm)atomic_inc (&mm->mm_users);task_unlock (task); + retval =0 ;if (!mm) - goto getlen_out; - -/* Check whether the mmaps could change if we sleep */ - volatile_task = (task != current ||atomic_read (&mm->mm_users) >2 ); - -/* decode f_pos */ - lineno = *ppos >>MAPS_LINE_SHIFT ; - column = *ppos & (MAPS_LINE_LENGTH -1 ); + goto out_free2; -/* quickly go to line lineno */ down_read (&mm->mmap_sem); -for (map = mm->mmap, i =0 ; map && (i < lineno); map = map->vm_next, i++) -continue ; - -for ( ; map ; map = next ) { -/* produce the next line */ -char *line; -char str[5 ], *cp = str; -int flags; -kdev_t dev; -unsigned long ino; -int maxlen = (sizeof (void *) ==4 ) ? -MAPS_LINE_MAX4 :MAPS_LINE_MAX8 ; + map = mm->mmap; + lineno =0 ; + loff =0 ; +if (count >PAGE_SIZE ) + count =PAGE_SIZE ; +while (map) {int len; - -/* -* Get the next vma now (but it won't be used if we sleep). -*/ - next = map->vm_next; - flags = map->vm_flags; - - *cp++ = flags &VM_READ ?'r' :'-' ; - *cp++ = flags &VM_WRITE ?'w' :'-' ; - *cp++ = flags &VM_EXEC ?'x' :'-' ; - *cp++ = flags &VM_MAYSHARE ?'s' :'p' ; - *cp++ =0 ; - - dev =0 ; - ino =0 ; -if (map->vm_file !=NULL ) { - dev = map->vm_file->f_dentry->d_inode->i_dev; - ino = map->vm_file->f_dentry->d_inode->i_ino; - line =d_path (map->vm_file->f_dentry, - map->vm_file->f_vfsmnt, - buffer,PAGE_SIZE ); - buffer[PAGE_SIZE -1 ] ='\n' ; - line -= maxlen; -if (line < buffer) - line = buffer; - }else - line = buffer; - - len =sprintf (line, -sizeof (void *) ==4 ?MAPS_LINE_FORMAT4 :MAPS_LINE_FORMAT8 , - map->vm_start, map->vm_end, str, map->vm_pgoff <<PAGE_SHIFT , -kdevname (dev), ino); - -if (map->vm_file) { -for (i = len; i < maxlen; i++) - line[i] =' ' ; - len = buffer +PAGE_SIZE - line; - }else - line[len++] ='\n' ; -if (column >= len) { - column =0 ;/* continue with next line at column 0 */ - lineno++; -continue ;/* we haven't slept */ +if (off >MAPS_LINE_LENGTH ) { + off -=MAPS_LINE_LENGTH ; + goto next; } - - i = len-column; -if (i > count) - i = count; -copy_to_user (destptr, line+column, i);/* may have slept */ - destptr += i; - count -= i; - column += i; -if (column >= len) { - column =0 ;/* next time: next line at column 0 */ - lineno++; + len =proc_pid_maps_get_line (tmp, map); + len -= off; +if (len >0 ) { +if (retval+len > count) { +/* only partial line transfer possible */ + len = count - retval; +/* save the offset where the next read +* must start */ + loff = len+off; + } +memcpy (kbuf+retval, tmp+off, len); + retval += len; } - -/* done? */ -if (count ==0 ) -break ; - -/* By writing to user space, we might have slept. -* Stop the loop, to avoid a race condition. -*/ -if (volatile_task) + off =0 ; +next: +if (!loff) + lineno++; +if (retval >= count)break ; +if (loff)BUG (); + map = map->vm_next; + } +if (retval > count)BUG (); +if (copy_to_user (buf, kbuf, retval)) { + retval = -EFAULT ; + }else { + *ppos = (lineno <<MAPS_LINE_SHIFT ) + loff; }up_read (&mm->mmap_sem); - -/* encode f_pos */ - *ppos = (lineno <<MAPS_LINE_SHIFT ) + column;mmput (mm); -getlen_out: - retval = destptr - buf; -free_page ((unsigned long )buffer); +out_free2: +free_page ((unsigned long )tmp); +out_free1: +free_page ((unsigned long )kbuf); out:return retval; }
Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:
- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.
- include/linux/mm.h: declaration for above function
- fs/exec.c: have do_coredump() call this function and not get the read lock around the binfmt coredumper
David
fs/exec.c +8 -2
@@ -947,6 +947,14 @@if (current->rlim[RLIMIT_CORE ].rlim_cur < binfmt->min_coredump) goto fail; +/* make sure the attached VM has a single ref (this process) to make +* sure only do_exit() will change the VMA list, so we don't have to +* lock the mm->sem around the binfmt coredumper +*/ + retval =copy_mm_for_coredump (current); +if (retval<0 ) + goto fail; +memcpy (corename,"core." ,5 ); corename[4 ] ='\0' ;if (core_uses_pid ||atomic_read (¤t->mm->mm_users) !=1 )
@@ -969,9 +977,7 @@if (do_truncate (file->f_dentry,0 ) !=0 ) goto close_fail; -down_read (¤t->mm->mmap_sem); retval = binfmt->core_dump (signr, regs, file); -up_read (¤t->mm->mmap_sem); close_fail:filp_close (file,NULL );
include/linux/mm.h +1 -0
@@ -615,6 +615,7 @@ }extern struct vm_area_struct *find_extend_vma (struct mm_struct *mm,unsigned long addr); +extern struct mm_struct *copy_mm_for_coredump (struct task_struct *tsk);#endif /* __KERNEL__ */
kernel/fork.c +49 -0
55 lines not on this page · raw message
Quoted 2 lines
Okay preliminary as-yet-untested patch to cure coredumping of the need> to hold the mm semaphore:
If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out
Okay tested patch to cure coredumping of the need to hold the mm semaphore:
- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.
- include/linux/mm.h: declaration for above function
- fs/exec.c: have do_coredump() call this function and not get the read lock around the binfmt coredumper
It works, and you can core dump without oopsing.
David
fs/exec.c +8 -2
@@ -947,6 +947,14 @@if (current->rlim[RLIMIT_CORE ].rlim_cur < binfmt->min_coredump) goto fail; +/* make sure the attached VM has a single ref (this process) to make +* sure only do_exit() will change the VMA list, so we don't have to +* lock the mm->sem around the binfmt coredumper +*/ + retval =copy_mm_for_coredump (current); +if (retval<0 ) + goto fail; +memcpy (corename,"core." ,5 ); corename[4 ] ='\0' ;if (core_uses_pid ||atomic_read (¤t->mm->mm_users) !=1 )
@@ -969,9 +977,7 @@if (do_truncate (file->f_dentry,0 ) !=0 ) goto close_fail; -down_read (¤t->mm->mmap_sem); retval = binfmt->core_dump (signr, regs, file); -up_read (¤t->mm->mmap_sem); close_fail:filp_close (file,NULL );
include/linux/mm.h +1 -0
@@ -615,6 +615,7 @@ }extern struct vm_area_struct *find_extend_vma (struct mm_struct *mm,unsigned long addr); +extern int copy_mm_for_coredump (struct task_struct *tsk);#endif /* __KERNEL__ */
kernel/fork.c +49 -0
55 lines not on this page · raw message
Quoted 6 lines
Quoted 3 lines
Okay preliminary as-yet-untested patch to cure coredumping of the need> to hold the mm semaphore:
If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out
Arjan said there was such a beast in the -ac stuff, but I guess this explains why I couldn't find it... Do you have any idea which -ac's?
Oh well, I'd already done my patch anyway. Does it look okay to you.
David
Quoted 5 lines
Quoted 2 lines
If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out
Arjan said there was such a beast in the -ac stuff, but I guess this explains why I couldn't find it... Do you have any idea which -ac's?
It went in with the threaded core dump patch
Quoted 6 lines · David Howells
Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:
-
kernel/fork.c: function to partially copy anmm_structand attach it to thetask_structin place of the old.
Oh, please no.
If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.
Linus
Looking through the do_page_fault(), I noticed there's a race in expand stack because expand_stack() expects the caller to have the mm-sem write-locked.
I've attached a patch that might fix it appropriately. Alternatively, it may be worth applying Andrea's 00_silent-stack-overflow-10 patch which fixes this and something else too.
David
include/linux/mm.h +5 -2
@@ -586,11 +586,11 @@* before relocating the vma range ourself. */ address &=PAGE_MASK ; +spin_lock (&vma->vm_mm->page_table_lock); grow = (vma->vm_start - address) >>PAGE_SHIFT ;if (vma->vm_end - address > current->rlim[RLIMIT_STACK ].rlim_cur || ((vma->vm_mm->total_vm + grow) <<PAGE_SHIFT ) > current->rlim[RLIMIT_AS ].rlim_cur) -return -ENOMEM ; -spin_lock (&vma->vm_mm->page_table_lock); + goto nomem; vma->vm_start = address; vma->vm_pgoff -= grow; vma->vm_mm->total_vm += grow;
@@ -598,6 +598,9 @@ vma->vm_mm->locked_vm += grow;spin_unlock (&vma->vm_mm->page_table_lock);return 0 ; + nomem: +spin_unlock (&vma->vm_mm->page_table_lock); +return -ENOMEM ; }/* Look up the first VMA which satisfies addr < vm_end, NULL if none. */
Quoted 7 lines · David Howells
Looking through the
do_page_fault(), I noticed there's a race in expand stack becauseexpand_stack()expects the caller to have the mm-sem write-locked.I've attached a patch that might fix it appropriately. Alternatively, it may be worth applying Andrea's 00_silent-stack-overflow-10 patch which fixes this and something else too.
Yep, it's here:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre11aa1/00_silent-stack-overflow-10
I also added the documentation on the locking on top of expand_stack.
My patch also enforced a gap of one page (sysctl configurable in with page granularity) between a growsdown vma and its previous vma, so that we can more easily trap stack overflows on the heap. (such part isn't related to the race fix but it was controversial but since it's quite useful too I didn't splitted it out :)
Andrea
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · David Howells
Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:
-
kernel/fork.c: function to partially copy anmm_structand attach it to thetask_structin place of the old.
Quoted 6 lines
Oh, please no.
If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.
But surely giving rw-semaphores this behaviour is even worse... It introduces the possibility of livelock, and so DoS attacks, and it affects more than just access to the mm_struct.
Also comparing them to rw-spinlocks isn't really fair IMHO since they have different restrictions. Things inside spinlocks aren't allowed to sleep, and mustn't incur pagefaults.
I also don't think the hack is that bad. All it's doing is taking a copy of the process's VM decription so that it knows that nobody is going to modify it whilst a coredump is in progress. Furthermore, it _only_ affects the coredump path, and the coredump path is just about the last thing on the agenda for a dying process.
However, if you don't like that, how about just changing the lock on mm_struct to a special mm_struct-only type lock that has a recursive lock operation for use by the pagefault handler (and _only_ the pagefault handler)? I've attached a patch to do just that. This introduces five operations:
- mm_lock_shared() - get shared lock fairly
- mm_lock_shared_recursive() - get shared lock unfairly
- mm_unlock_shared() - release shared lock
- mm_lock_exclusive() - get exclusive lock
- mm_unlock_exclusive() - release exclusive lock
David
arch/alpha/kernel/osf_sys.c +2 -2
@@ -241,9 +241,9 @@ goto out; } flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_mmap (file, addr, len, prot, flags, off); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file); out:
arch/alpha/mm/fault.c +4 -4
@@ -113,7 +113,7 @@ goto vmalloc_fault;#endif -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -147,7 +147,7 @@* the fault. */ fault =handle_mm_fault (mm, vma, address, cause >0 ); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (fault <0 ) goto out_of_memory;
@@ -161,7 +161,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (user_mode (regs)) {force_sig (SIGSEGV , current);
@@ -198,7 +198,7 @@if (current->pid ==1 ) { current->policy |=SCHED_YIELD ;schedule (); -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); goto survive; } printk(KERN_ALERT "VM: killing process %s(%d)\n",
arch/arm/kernel/sys_arm.c +4 -4
@@ -74,9 +74,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
@@ -125,9 +125,9 @@ vectors_base() == 0) goto out; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_mremap (addr, old_len, new_len, flags, new_addr); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out:return ret;
arch/arm/mm/fault-common.c +2 -2
@@ -251,9 +251,9 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); fault =__do_page_fault (mm, addr, error_code, tsk); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Handle the "normal" case first
arch/cris/kernel/sys_cris.c +2 -2
@@ -59,9 +59,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/cris/mm/fault.c +5 -5
@@ -266,7 +266,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -324,7 +324,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -334,7 +334,7 @@ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); bad_area_nosemaphore:DPG (show_registers (regs));
@@ -397,14 +397,14 @@*/ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (user_mode (regs))do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/i386/kernel/ldt.c +2 -2
@@ -73,7 +73,7 @@* the GDT index of the LDT is allocated dynamically, and is * limited by MAX_LDT_DESCRIPTORS. */ -down_write (&mm->mmap_sem); +mm_lock_exclusive (mm);if (!mm->context.segments) {void * segments =vmalloc (LDT_ENTRIES *LDT_ENTRY_SIZE ); error = -ENOMEM ;
@@ -124,7 +124,7 @@ error =0 ; out_unlock: -up_write (&mm->mmap_sem); +mm_unlock_exclusive (mm); out:return error; }
arch/i386/kernel/sys_i386.c +2 -2
@@ -55,9 +55,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/i386/mm/fault.c +6 -6
@@ -191,7 +191,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma)
@@ -265,7 +265,7 @@if (bit <32 ) tsk->thread.screen_bitmap |=1 << bit; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -273,7 +273,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* User mode accesses just cause a SIGSEGV */ if (error_code &4 ) {
@@ -341,11 +341,11 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (tsk->pid ==1 ) { tsk->policy |=SCHED_YIELD ;schedule (); -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); goto survive; }printk ("VM: killing process %s\n" , tsk->comm);
@@ -354,7 +354,7 @@ goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/ia64/ia32/sys_ia32.c +8 -8
@@ -245,9 +245,9 @@ }__copy_user (back, (char *)addr + len,PAGE_SIZE - ((addr + len) & ~PAGE_MASK )); } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); r =do_mmap (0 , baddr, len + (addr - baddr), prot, flags |MAP_ANONYMOUS ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (r <0 )return (r);if (addr ==0 )
@@ -291,9 +291,9 @@ poff = offset &PAGE_MASK ; len += offset - poff; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, poff >>PAGE_SHIFT ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (!IS_ERR ((void *) error)) error += offset - poff;
@@ -338,9 +338,9 @@if ((a.offset & ~PAGE_MASK ) !=0 )return -EINVAL ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap_pgoff (file, a.addr, a.len, a.prot, a.flags, a.offset >>PAGE_SHIFT ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);#else retval =ia32_do_mmap (file, a.addr, a.len, a.prot, a.flags, a.fd, a.offset);#endif
@@ -2605,11 +2605,11 @@return (-EFAULT ); } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); addr =do_mmap_pgoff (file,IA32_IOBASE ,IOLEN ,PROT_READ |PROT_WRITE ,MAP_SHARED , (ia64_iobase & ~PAGE_OFFSET ) >>PAGE_SHIFT ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (addr >=0 ) {ia64_set_kr (IA64_KR_IO_BASE , addr);
arch/ia64/kernel/sys_ia64.c +4 -4
@@ -106,7 +106,7 @@* check and the clearing of r8. However, we can't call sys_brk() because we need * to acquire the mmap_sem before we can do the test... */ -down_write (&mm->mmap_sem); +mm_lock_exclusive (mm);if (brk < mm->end_code) goto out;
@@ -146,7 +146,7 @@ mm->brk = brk; out: retval = mm->brk; -up_write (&mm->mmap_sem); +mm_unlock_exclusive (mm); regs->r8 =0 ;/* ensure large retval isn't mistaken as error code */ return retval; }
@@ -205,9 +205,9 @@if (rgn_index (addr) !=rgn_index (addr + len))return -EINVAL ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); addr =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/ia64/mm/fault.c +4 -4
@@ -60,7 +60,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma_prev (mm, address, &prev_vma);if (!vma)
@@ -112,7 +112,7 @@default : goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ; check_expansion:
@@ -135,7 +135,7 @@ goto good_area; bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (isr &IA64_ISR_SP ) {/* * This fault was due to a speculative load set the "ed" bit in the psr to
@@ -184,7 +184,7 @@return ; out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , current->comm);if (user_mode (regs))do_exit (SIGKILL );
arch/m68k/kernel/sys_m68k.c +4 -4
@@ -59,9 +59,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
@@ -146,9 +146,9 @@ } a.flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, a.addr, a.len, a.prot, a.flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file); out:
arch/m68k/mm/fault.c +3 -3
@@ -101,7 +101,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma)
@@ -168,7 +168,7 @@#warning should be obsolete now...if (CPU_IS_040_OR_060 )flush_tlb_page (vma, address); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return 0 ;/*
@@ -203,6 +203,6 @@ current->thread.faddr = address; send_sig: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return send_fault_sig (regs); }
arch/mips/kernel/irixelf.c +10 -10
@@ -314,12 +314,12 @@ (unsigned long ) elf_prot, (unsigned long ) elf_type, (unsigned long) (eppnt->p_offset & 0xfffff000));#endif -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (interpreter, vaddr, eppnt->p_filesz + (eppnt->p_vaddr &0xfff ), elf_prot, elf_type, eppnt->p_offset &0xfffff000 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error <0 && error >-1024 ) {printk ("Aieee IRIX interp mmap error=%d\n" , error);
@@ -498,12 +498,12 @@ prot = (epp->p_flags &PF_R ) ?PROT_READ :0 ; prot |= (epp->p_flags &PF_W ) ?PROT_WRITE :0 ; prot |= (epp->p_flags &PF_X ) ?PROT_EXEC :0 ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); (void )do_mmap (fp, (epp->p_vaddr &0xfffff000 ), (epp->p_filesz + (epp->p_vaddr &0xfff )), prot,EXEC_MAP_FLAGS , (epp->p_offset &0xfffff000 )); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);/* Fixup location tracking vars. */ if ((epp->p_vaddr &0xfffff000 ) < *estack)
@@ -762,10 +762,10 @@* Since we do not have the power to recompile these, we * emulate the SVr4 behavior. Sigh. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); (void )do_mmap (NULL ,0 ,4096 ,PROT_READ |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);#endif start_thread (regs, elf_entry, bprm->p);
@@ -837,14 +837,14 @@while (elf_phdata->p_type !=PT_LOAD ) elf_phdata++;/* Now use mmap to map the library into memory. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (file, elf_phdata->p_vaddr &0xfffff000 , elf_phdata->p_filesz + (elf_phdata->p_vaddr &0xfff ),PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE , elf_phdata->p_offset &0xfffff000 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); k = elf_phdata->p_vaddr + elf_phdata->p_filesz;if (k > elf_bss) elf_bss = k;
@@ -916,12 +916,12 @@ prot = (hp->p_flags &PF_R ) ?PROT_READ :0 ; prot |= (hp->p_flags &PF_W ) ?PROT_WRITE :0 ; prot |= (hp->p_flags &PF_X ) ?PROT_EXEC :0 ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap (filp, (hp->p_vaddr &0xfffff000 ), (hp->p_filesz + (hp->p_vaddr &0xfff )), prot, (MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE ), (hp->p_offset &0xfffff000 )); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (retval != (hp->p_vaddr &0xfffff000 )) {printk ("irix_mapelf: do_mmap fails with %d!\n" , retval);
arch/mips/kernel/syscall.c +2 -2
@@ -69,9 +69,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/mips/kernel/sysirix.c +8 -8
@@ -471,7 +471,7 @@if (retval)return retval; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); pgdp =pgd_offset (mm, addr); pmdp =pmd_offset (pgdp, addr); ptep =pte_offset (pmdp, addr);
@@ -484,7 +484,7 @@ PAGE_SHIFT, pageno); } } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);break ; }
@@ -534,7 +534,7 @@struct mm_struct *mm = current->mm;int ret; -down_write (&mm->mmap_sem); +mm_lock_exclusive (mm);if (brk < mm->end_code) { ret = -ENOMEM ; goto out;
@@ -592,7 +592,7 @@ ret =0 ; out: -up_write (&mm->mmap_sem); +mm_unlock_exclusive (mm);return ret; }
@@ -1082,9 +1082,9 @@ flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap (file, addr, len, prot, flags, offset); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
@@ -1642,9 +1642,9 @@ flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/mips/mm/fault.c +5 -5
@@ -72,7 +72,7 @@printk ("[%s:%d:%08lx:%ld:%08lx]\n" , current->comm, current->pid, address, write, regs->cp0_epc);#endif -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -115,7 +115,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -123,7 +123,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -177,14 +177,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (user_mode (regs))do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/mips64/kernel/linux32.c +2 -2
@@ -443,10 +443,10 @@* `execve' frees all current memory we only have to do an * `munmap' if the `execve' failes. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); av = (char **)do_mmap_pgoff (0 ,0 , len,PROT_READ |PROT_WRITE ,MAP_PRIVATE |MAP_ANONYMOUS ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (IS_ERR (av))return (long ) av;
arch/mips64/kernel/syscall.c +2 -2
@@ -65,9 +65,9 @@ } flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (file, addr, len, prot, flags, offset); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file); out:
arch/mips64/mm/fault.c +5 -5
@@ -124,7 +124,7 @@printk ("Cpu%d[%s:%d:%08lx:%ld:%08lx]\n" ,smp_processor_id (), current->comm, current->pid, address, write, regs->cp0_epc);#endif -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -167,7 +167,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -175,7 +175,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); bad_area_nosemaphore:if (user_mode (regs)) {
@@ -233,14 +233,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (user_mode (regs))do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/parisc/kernel/sys_parisc.c +2 -2
@@ -51,7 +51,7 @@struct file * file =NULL ;int error; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);lock_kernel ();if (!(flags &MAP_ANONYMOUS )) { error = -EBADF ;
@@ -65,7 +65,7 @@fput (file); out:unlock_kernel (); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return error; }
arch/parisc/mm/fault.c +4 -4
@@ -175,7 +175,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =pa_find_vma (mm, address);if (!vma) goto bad_area;
@@ -218,14 +218,14 @@default : goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/* * Something tried to access memory that isn't in our memory map.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (user_mode (regs)) {struct siginfo si;
@@ -275,7 +275,7 @@parisc_terminate ("Bad Address (null pointer deref?)" ,regs,code,address); out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , current->comm);if (user_mode (regs))do_exit (SIGKILL );
arch/ppc/kernel/syscalls.c +2 -2
@@ -202,9 +202,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file); out:
arch/ppc/mm/fault.c +5 -5
@@ -103,7 +103,7 @@bad_page_fault (regs, address,SIGSEGV );return ; } -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -163,7 +163,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * keep track of tlb+htab misses that are good addrs but * just need pte's created via handle_mm_fault()
@@ -173,7 +173,7 @@return ; bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); pte_errors++;/* User mode accesses cause a SIGSEGV */
@@ -194,7 +194,7 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , current->comm);if (user_mode (regs))do_exit (SIGKILL );
@@ -202,7 +202,7 @@return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); info.si_signo =SIGBUS ; info.si_errno =0 ; info.si_code =BUS_ADRERR ;
arch/s390/kernel/sys_s390.c +2 -2
@@ -61,9 +61,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/s390/mm/fault.c +5 -5
@@ -113,7 +113,7 @@* task's user address space, so we search the VMAs */ -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma)
@@ -164,7 +164,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -172,7 +172,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* User mode accesses just cause a SIGSEGV */ if (regs->psw.mask &PSW_PROBLEM_STATE ) {
@@ -231,14 +231,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (regs->psw.mask &PSW_PROBLEM_STATE )do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/s390x/kernel/binfmt_elf32.c +2 -2
@@ -194,11 +194,11 @@if (!addr) addr =0x40000000 ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); map_addr =do_mmap (filep,ELF_PAGESTART (addr), eppnt->p_filesz +ELF_PAGEOFFSET (eppnt->p_vaddr), prot, type, eppnt->p_offset -ELF_PAGEOFFSET (eppnt->p_vaddr)); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return (map_addr); }
arch/s390x/kernel/exec32.c +2 -2
@@ -54,7 +54,7 @@if (!mpnt)return -ENOMEM ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); { mpnt->vm_mm = current->mm; mpnt->vm_start =PAGE_MASK & (unsigned long ) bprm->p;
@@ -77,7 +77,7 @@ } stack_base +=PAGE_SIZE ; } -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return 0 ; }
arch/s390x/kernel/linux32.c +2 -2
@@ -4186,14 +4186,14 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff);if (!IS_ERR ((void *) error) && error + len >=0x80000000ULL ) {/* Result is out of bounds. */ do_munmap (current->mm, addr, len); error = -ENOMEM ; } -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/s390x/kernel/sys_s390.c +2 -2
@@ -61,9 +61,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/s390x/mm/fault.c +5 -5
@@ -141,7 +141,7 @@* task's user address space, so we search the VMAs */ -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma) {
@@ -195,7 +195,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -203,7 +203,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* User mode accesses just cause a SIGSEGV */ if (regs->psw.mask &PSW_PROBLEM_STATE ) {
@@ -262,14 +262,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (regs->psw.mask &PSW_PROBLEM_STATE )do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/sh/kernel/sys_sh.c +2 -2
@@ -96,9 +96,9 @@ goto out; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (file)fput (file);
arch/sh/mm/fault.c +5 -5
@@ -105,7 +105,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared_recursive (mm); vma =find_vma (mm, address);if (!vma)
@@ -147,7 +147,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -155,7 +155,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);if (user_mode (regs)) { tsk->thread.address = address;
@@ -204,14 +204,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (user_mode (regs))do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/sparc/kernel/sys_sparc.c +4 -4
@@ -242,9 +242,9 @@ flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap_pgoff (file, addr, len, prot, flags, pgoff); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out_putf:if (file)
@@ -288,7 +288,7 @@if (old_len >TASK_SIZE -PAGE_SIZE || new_len >TASK_SIZE -PAGE_SIZE ) goto out; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (flags &MREMAP_FIXED ) { if (ARCH_SUN4C_SUN4 && new_addr < 0xe0000000 &&
@@ -323,7 +323,7 @@ } ret =do_mremap (addr, old_len, new_len, flags, new_addr); out_sem: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out:return ret; }
arch/sparc/kernel/sys_sunos.c +4 -4
@@ -116,9 +116,9 @@ } flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap (file, addr, len, prot, flags, off); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (!ret_type) retval = ((retval <PAGE_OFFSET ) ?0 : retval);
@@ -145,7 +145,7 @@unsigned long rlim;unsigned long newbrk, oldbrk; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (ARCH_SUN4C_SUN4 ) {if (brk >=0x20000000 && brk <0xe0000000 ) { goto out;
@@ -208,7 +208,7 @@do_brk (oldbrk, newbrk-oldbrk); retval =0 ; out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return retval; }
arch/sparc/mm/fault.c +9 -9
@@ -222,7 +222,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +mm_lock_shared (mm);/* * The kernel referencing a bad kernel pointer can lock up
@@ -272,7 +272,7 @@default : goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ;/*
@@ -280,7 +280,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -336,14 +336,14 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , tsk->comm);if (from_user)do_exit (SIGKILL ); goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); info.si_signo =SIGBUS ; info.si_errno =0 ; info.si_code =BUS_ADRERR ;
@@ -477,7 +477,7 @@printk ("wf<pid=%d,wr=%d,addr=%08lx>\n" , tsk->pid, write, address);#endif -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -498,10 +498,10 @@ }if (!handle_mm_fault (mm, vma, address, write)) goto do_sigbus; -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return ; bad_area: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);#if 0 printk ("Window whee %s [%d]: segfaults at %08lx\n" , tsk->comm, tsk->pid, address);
@@ -516,7 +516,7 @@return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); info.si_signo =SIGBUS ; info.si_errno =0 ; info.si_code =BUS_ADRERR ;
arch/sparc64/kernel/binfmt_aout32.c +6 -6
@@ -277,24 +277,24 @@ goto beyond_if; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (bprm->file,N_TXTADDR (ex), ex.a_text,PROT_READ |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , fd_offset); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error !=N_TXTADDR (ex)) {send_sig (SIGKILL , current,0 );return error; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (bprm->file,N_DATADDR (ex), ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , fd_offset + ex.a_text); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error !=N_DATADDR (ex)) {send_sig (SIGKILL , current,0 );return error;
@@ -369,12 +369,12 @@ start_addr = ex.a_entry &0xfffff000 ;/* Now use mmap to map the library into memory. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (file, start_addr, ex.a_text + ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE ,N_TXTOFF (ex)); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); retval = error;if (error != start_addr) goto out;
arch/sparc64/kernel/sys_sparc.c +6 -6
@@ -292,9 +292,9 @@ goto out_putf; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap (file, addr, len, prot, flags, off); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out_putf:if (file)
@@ -310,9 +310,9 @@if (len > -PAGE_OFFSET || (addr <PAGE_OFFSET && addr + len > -PAGE_OFFSET ))return -EINVAL ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_munmap (current->mm, addr, len); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return ret; }
@@ -332,7 +332,7 @@ goto out;if (addr <PAGE_OFFSET && addr + old_len > -PAGE_OFFSET ) goto out; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (flags &MREMAP_FIXED ) {if (new_addr <PAGE_OFFSET && new_addr + new_len > -PAGE_OFFSET )
@@ -363,7 +363,7 @@ } ret =do_mremap (addr, old_len, new_len, flags, new_addr); out_sem: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out:return ret; }
arch/sparc64/kernel/sys_sparc32.c +2 -2
@@ -4141,7 +4141,7 @@ goto out;if (addr >0xf0000000UL - old_len) goto out; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (flags &MREMAP_FIXED ) {if (new_addr >0xf0000000UL - new_len) goto out_sem;
@@ -4171,7 +4171,7 @@ } ret =do_mremap (addr, old_len, new_len, flags, new_addr); out_sem: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); out:return ret; }
arch/sparc64/kernel/sys_sunos32.c +4 -4
@@ -100,12 +100,12 @@ flags &= ~_MAP_NEW; flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); retval =do_mmap (file, (unsigned long ) addr, (unsigned long ) len, (unsigned long ) prot, (unsigned long ) flags, (unsigned long ) off); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (!ret_type) retval = ((retval <0xf0000000 ) ?0 : retval); out_putf:
@@ -126,7 +126,7 @@unsigned long rlim;unsigned long newbrk, oldbrk, brk = (unsigned long ) baddr; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (brk < current->mm->end_code) goto out; newbrk =PAGE_ALIGN (brk);
@@ -170,7 +170,7 @@do_brk (oldbrk, newbrk-oldbrk); retval =0 ; out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return retval; }
arch/sparc64/mm/fault.c +5 -5
@@ -306,7 +306,7 @@ address &=0xffffffff ; } -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -378,7 +378,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); goto fault_done;/*
@@ -387,7 +387,7 @@*/ bad_area: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); handle_kernel_fault:do_kernel_fault (regs, si_code, fault_code, insn, address);
@@ -400,7 +400,7 @@*/ out_of_memory: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);printk ("VM: killing process %s\n" , current->comm);if (!(regs->tstate &TSTATE_PRIV ))do_exit (SIGKILL );
@@ -412,7 +412,7 @@ do_sigbus: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* * Send a sigbus, regardless of whether we were in kernel
arch/sparc64/solaris/misc.c +2 -2
@@ -92,12 +92,12 @@ ret_type = flags & _MAP_NEW; flags &= ~_MAP_NEW; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); retval =do_mmap (file, (unsigned long ) addr, (unsigned long ) len, (unsigned long ) prot, (unsigned long ) flags, off); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (!ret_type) retval = ((retval <0xf0000000 ) ?0 : retval);
drivers/char/mem.c +3 -3
@@ -350,7 +350,7 @@ mm = current->mm;/* Oops, this was forgotten before. -ben */ -down_read (&mm->mmap_sem); +mm_lock_shared (mm);/* For private mappings, just map in zero pages. */ for (vma =find_vma (mm, addr); vma; vma = vma->vm_next) {
@@ -374,7 +374,7 @@ goto out_up; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* The shared case is hard. Let's do the conventional zeroing. */ do {
@@ -389,7 +389,7 @@return size; out_up: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return size; }
drivers/sgi/char/graphics.c +2 -2
@@ -152,11 +152,11 @@* sgi_graphics_mmap */ disable_gconsole (); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); r =do_mmap (file, (unsigned long )vaddr, cards[board].g_regs_size,PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (r)return r; }
drivers/sgi/char/shmiq.c +2 -2
@@ -285,11 +285,11 @@ s = req.arg *sizeof (struct shmqevent ) +sizeof (struct sharedMemoryInputQueue ); v =sys_munmap (vaddr, s); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);do_munmap (current->mm, vaddr, s);do_mmap (filp, vaddr, s,PROT_READ |PROT_WRITE ,MAP_PRIVATE |MAP_FIXED ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); shmiqs[minor].events = req.arg; shmiqs[minor].mapped =1 ;
fs/binfmt_aout.c +6 -6
@@ -377,24 +377,24 @@ goto beyond_if; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (bprm->file,N_TXTADDR (ex), ex.a_text,PROT_READ |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , fd_offset); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error !=N_TXTADDR (ex)) {send_sig (SIGKILL , current,0 );return error; } -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (bprm->file,N_DATADDR (ex), ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , fd_offset + ex.a_text); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error !=N_DATADDR (ex)) {send_sig (SIGKILL , current,0 );return error;
@@ -476,12 +476,12 @@ goto out; }/* Now use mmap to map the library into memory. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (file, start_addr, ex.a_text + ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE ,N_TXTOFF (ex)); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); retval = error;if (error != start_addr) goto out;
fs/binfmt_elf.c +6 -6
@@ -224,11 +224,11 @@ {unsigned long map_addr; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); map_addr =do_mmap (filep,ELF_PAGESTART (addr), eppnt->p_filesz +ELF_PAGEOFFSET (eppnt->p_vaddr), prot, type, eppnt->p_offset -ELF_PAGEOFFSET (eppnt->p_vaddr)); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return (map_addr); }
@@ -743,10 +743,10 @@Since we do not have the power to recompile these, we emulate the SVr4 behavior. Sigh. */ /* N.B. Shouldn't the size here be PAGE_SIZE?? */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error =do_mmap (NULL ,0 ,4096 ,PROT_READ |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE ,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm); }#ifdef ELF_PLAT_INIT
@@ -827,7 +827,7 @@while (elf_phdata->p_type !=PT_LOAD ) elf_phdata++;/* Now use mmap to map the library into memory. */ -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); error = do_mmap(file, ELF_PAGESTART(elf_phdata->p_vaddr), (elf_phdata->p_filesz +
@@ -836,7 +836,7 @@MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE , (elf_phdata->p_offset - ELF_PAGEOFFSET(elf_phdata->p_vaddr))); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);if (error !=ELF_PAGESTART (elf_phdata->p_vaddr)) goto out_free_ph;
fs/exec.c +4 -4
@@ -307,7 +307,7 @@if (!mpnt)return -ENOMEM ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); { mpnt->vm_mm = current->mm; mpnt->vm_start =PAGE_MASK & (unsigned long ) bprm->p;
@@ -330,7 +330,7 @@ } stack_base +=PAGE_SIZE ; } -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return 0 ; }
@@ -969,9 +969,9 @@if (do_truncate (file->f_dentry,0 ) !=0 ) goto close_fail; -down_read (¤t->mm->mmap_sem); +mm_lock_shared (current->mm); retval = binfmt->core_dump (signr, regs, file); -up_read (¤t->mm->mmap_sem); +mm_unlock_shared (current->mm); close_fail:filp_close (file,NULL );
fs/proc/array.c +8 -8
@@ -181,7 +181,7 @@unsigned long data =0 , stack =0 ;unsigned long exec =0 , lib =0 ; -down_read (&mm->mmap_sem); +mm_lock_shared (mm);for (vma = mm->mmap; vma; vma = vma->vm_next) {unsigned long len = (vma->vm_end - vma->vm_start) >>10 ;if (!vma->vm_file) {
@@ -212,7 +212,7 @@ mm->rss << (PAGE_SHIFT -10 ), data - stack, stack, exec - lib, lib); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);return buffer; }
@@ -317,7 +317,7 @@task_unlock (task);if (mm) {struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma = mm->mmap;while (vma) { vsize += vma->vm_end - vma->vm_start;
@@ -325,7 +325,7 @@ } eip =KSTK_EIP (task); esp =KSTK_ESP (task); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm); } wchan =get_wchan (task);
@@ -479,7 +479,7 @@task_unlock (task);if (mm) {struct vm_area_struct * vma; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma = mm->mmap;while (vma) {pgd_t *pgd =pgd_offset (mm, vma->vm_start);
@@ -500,7 +500,7 @@ drs += pages; vma = vma->vm_next; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);mmput (mm); } return sprintf(buffer,"%d %d %d %d %d %d %d\n",
@@ -577,7 +577,7 @@ column = *ppos & (MAPS_LINE_LENGTH -1 );/* quickly go to line lineno */ -down_read (&mm->mmap_sem); +mm_lock_shared (mm);for (map = mm->mmap, i =0 ; map && (i < lineno); map = map->vm_next, i++)continue ;
@@ -658,7 +658,7 @@if (volatile_task)break ; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);/* encode f_pos */ *ppos = (lineno <<MAPS_LINE_SHIFT ) + column;
fs/proc/base.c +2 -2
@@ -64,7 +64,7 @@task_unlock (task);if (!mm) goto out; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma = mm->mmap;while (vma) { if ((vma->vm_flags & VM_EXECUTABLE) &&
@@ -76,7 +76,7 @@ } vma = vma->vm_next; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);mmput (mm); out:return result;
include/linux/sched.h +67 -10
89 lines not on this page · raw message
ipc/shm.c +4 -4
@@ -619,9 +619,9 @@ shp->shm_nattch++;shm_unlock (shmid); -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); user_addr = (void *)do_mmap (file, addr, file->f_dentry->d_inode->i_size, prot, flags,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);down (&shm_ids.sem);if (!(shp =shm_lock (shmid)))
@@ -650,14 +650,14 @@struct mm_struct *mm = current->mm;struct vm_area_struct *shmd, *shmdnext; -down_write (&mm->mmap_sem); +mm_lock_exclusive (mm);for (shmd = mm->mmap; shmd; shmd = shmdnext) { shmdnext = shmd->vm_next;if (shmd->vm_ops == &shm_vm_ops && shmd->vm_start - (shmd->vm_pgoff <<PAGE_SHIFT ) == (ulong ) shmaddr)do_munmap (mm, shmd->vm_start, shmd->vm_end - shmd->vm_start); } -up_write (&mm->mmap_sem); +mm_unlock_exclusive (mm);return 0 ; }
kernel/acct.c +2 -2
@@ -315,13 +315,13 @@ vsize =0 ;if (current->mm) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +mm_lock_shared (current->mm); vma = current->mm->mmap;while (vma) { vsize += vma->vm_end - vma->vm_start; vma = vma->vm_next; } -up_read (¤t->mm->mmap_sem); +mm_unlock_shared (current->mm); } vsize = vsize /1024 ; ac.ac_mem =encode_comp_t (vsize);
kernel/fork.c +5 -3
@@ -216,7 +216,9 @@ {atomic_set (&mm->mm_users,1 );atomic_set (&mm->mm_count,1 ); -init_rwsem (&mm->mmap_sem); +spin_lock_init (&mm->mmsem_lock); + mm->mmsem_activity =0 ; +INIT_LIST_HEAD (&mm->mmsem_waiters); mm->page_table_lock =SPIN_LOCK_UNLOCKED ; mm->pgd =pgd_alloc (mm);if (mm->pgd)
@@ -333,9 +335,9 @@if (!mm_init (mm)) goto fail_nomem; -down_write (&oldmm->mmap_sem); +mm_lock_exclusive (oldmm); retval =dup_mmap (mm); -up_write (&oldmm->mmap_sem); +mm_unlock_exclusive (oldmm);if (retval) goto free_pt;
kernel/ksyms.c +2 -0
@@ -87,6 +87,8 @@EXPORT_SYMBOL (exit_files);EXPORT_SYMBOL (exit_fs);EXPORT_SYMBOL (exit_sighand); +EXPORT_SYMBOL (__mm_lock_wait); +EXPORT_SYMBOL (__mm_lock_wake);/* internal kernel memory management */ EXPORT_SYMBOL (_alloc_pages);
kernel/ptrace.c +2 -2
@@ -208,13 +208,13 @@if (!mm)return 0 ; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); vma =find_extend_vma (mm, addr); copied =0 ;if (vma) copied =access_mm (mm, vma, addr, buf, len, write); -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);mmput (mm);return copied; }
mm/filemap.c +6 -6
@@ -1949,7 +1949,7 @@struct vm_area_struct * vma;int unmapped_error, error = -EINVAL ; -down_read (¤t->mm->mmap_sem); +mm_lock_shared (current->mm);if (start & ~PAGE_MASK ) goto out; len = (len + ~PAGE_MASK ) &PAGE_MASK ;
@@ -1995,7 +1995,7 @@ vma = vma->vm_next; } out: -up_read (¤t->mm->mmap_sem); +mm_unlock_shared (current->mm);return error; }
@@ -2298,7 +2298,7 @@int unmapped_error =0 ;int error = -EINVAL ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (start & ~PAGE_MASK ) goto out;
@@ -2349,7 +2349,7 @@ } out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return error; }
@@ -2451,7 +2451,7 @@int unmapped_error =0 ;long error = -EINVAL ; -down_read (¤t->mm->mmap_sem); +mm_lock_shared (current->mm);if (start & ~PAGE_CACHE_MASK ) goto out;
@@ -2503,7 +2503,7 @@ } out: -up_read (¤t->mm->mmap_sem); +mm_unlock_shared (current->mm);return error; }
mm/memory.c +3 -3
@@ -464,7 +464,7 @@if (err)return err; -down_read (&mm->mmap_sem); +mm_lock_shared (mm); err = -EFAULT ; iobuf->locked =0 ;
@@ -522,12 +522,12 @@ ptr +=PAGE_SIZE ; } -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);dprintk ("map_user_kiobuf: end OK\n" );return 0 ; out_unlock: -up_read (&mm->mmap_sem); +mm_unlock_shared (mm);unmap_kiobuf (iobuf);dprintk ("map_user_kiobuf: end %d\n" , err);return err;
mm/mlock.c +8 -8
@@ -198,7 +198,7 @@unsigned long lock_limit;int error = -ENOMEM ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); len =PAGE_ALIGN (len + (start & ~PAGE_MASK )); start &=PAGE_MASK ;
@@ -219,7 +219,7 @@ error =do_mlock (start, len,1 ); out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return error; }
@@ -227,11 +227,11 @@ {int ret; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); len =PAGE_ALIGN (len + (start & ~PAGE_MASK )); start &=PAGE_MASK ; ret =do_mlock (start, len,0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return ret; }
@@ -268,7 +268,7 @@unsigned long lock_limit;int ret = -EINVAL ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm);if (!flags || (flags & ~(MCL_CURRENT |MCL_FUTURE ))) goto out;
@@ -286,7 +286,7 @@ ret =do_mlockall (flags); out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return ret; }
@@ -294,8 +294,8 @@ {int ret; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_mlockall (0 ); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return ret; }
mm/mmap.c +86 -4
113 lines not on this page · raw message
mm/mprotect.c +2 -2
@@ -281,7 +281,7 @@if (end == start)return 0 ; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); vma =find_vma_prev (current->mm, start, &prev); error = -EFAULT ;
@@ -332,6 +332,6 @@ prev->vm_mm->map_count--; } out: -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return error; }
mm/mremap.c +2 -2
@@ -346,8 +346,8 @@ {unsigned long ret; -down_write (¤t->mm->mmap_sem); +mm_lock_exclusive (current->mm); ret =do_mremap (addr, old_len, new_len, flags, new_addr); -up_write (¤t->mm->mmap_sem); +mm_unlock_exclusive (current->mm);return ret; }
Here's a patch to make rwsems unfair.
David
arch/alpha/config.in +0 -2
@@ -5,8 +5,6 @@ define_bool CONFIG_ALPHA y define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y mainmenu_name "Kernel configuration of Linux for Alpha machines"
arch/arm/config.in +0 -2
@@ -9,8 +9,6 @@ define_bool CONFIG_SBUS n define_bool CONFIG_MCA n define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment
arch/arm/def-configs/anakin +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/assabet +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/bitsy +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/ebsa110 +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/flexanet +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/integrator +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/lart +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/neponset +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/pleb +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/rpc +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/arm/def-configs/shark +0 -2
@@ -6,8 +6,6 @@ # CONFIG_SBUS is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/cris/config.in +0 -2
@@ -5,8 +5,6 @@ mainmenu_name "Linux/CRIS Kernel Configuration" define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/cris/defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # # Code maturity level options
arch/i386/config.in +0 -4
@@ -50,8 +50,6 @@ define_bool CONFIG_X86_CMPXCHG n define_bool CONFIG_X86_XADD n define_int CONFIG_X86_L1_CACHE_SHIFT 4 - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n else define_bool CONFIG_X86_WP_WORKS_OK y define_bool CONFIG_X86_INVLPG y
@@ -59,8 +57,6 @@ define_bool CONFIG_X86_XADD y define_bool CONFIG_X86_BSWAP y define_bool CONFIG_X86_POPAD_OK y - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y fi if [ "$CONFIG_M486" = "y" ]; then define_int CONFIG_X86_L1_CACHE_SHIFT 4
arch/i386/defconfig +0 -2
@@ -42,8 +42,6 @@ CONFIG_X86_XADD=y CONFIG_X86_BSWAP=y CONFIG_X86_POPAD_OK=y -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_X86_L1_CACHE_SHIFT=5 CONFIG_X86_TSC=y CONFIG_X86_GOOD_APIC=y
arch/ia64/config.in +0 -2
@@ -23,8 +23,6 @@ define_bool CONFIG_EISA n define_bool CONFIG_MCA n define_bool CONFIG_SBUS n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -2
@@ -4,8 +4,6 @@ # define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux/68k Kernel Configuration"
arch/mips/config.in +0 -2
@@ -68,8 +68,6 @@ fi bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000 -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n # # Select some configuration options automatically for certain systems.
arch/mips/defconfig +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_ARC32=y
arch/mips/defconfig-atlas +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_PCI=y
arch/mips/defconfig-ddb5476 +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_ISA=y
arch/mips/defconfig-ddb5477 +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_CPU_LITTLE_ENDIAN=y
arch/mips/defconfig-decstation +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set # CONFIG_ISA is not set
arch/mips/defconfig-ip22 +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_ARC32=y
arch/mips/defconfig-it8172 +0 -2
@@ -37,8 +37,6 @@ # CONFIG_IT8172_SCR1 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_PCI=y
arch/mips/defconfig-malta +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_I8259=y
arch/mips/defconfig-nino +0 -2
@@ -35,8 +35,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_PC_KEYB=y
arch/mips/defconfig-ocelot +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_PCI=y
arch/mips/defconfig-pb1000 +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set CONFIG_MIPS_PB1000=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_MIPS_AU1000=y
arch/mips/defconfig-rm200 +0 -2
@@ -32,8 +32,6 @@ # CONFIG_MIPS_ITE8172 is not set # CONFIG_MIPS_IVR is not set # CONFIG_MIPS_PB1000 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set CONFIG_ARC32=y
arch/mips64/config.in +0 -2
@@ -27,8 +27,6 @@ fi endmenu -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n # # Select some configuration options automatically based on user selections
arch/mips64/defconfig +0 -2
@@ -19,8 +19,6 @@ # CONFIG_REPLICATE_KTEXT is not set # CONFIG_REPLICATE_EXHANDLERS is not set CONFIG_SMP=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_BOOT_ELF64=y CONFIG_ARC64=y CONFIG_COHERENT_IO=y
arch/mips64/defconfig-ip22 +0 -2
@@ -12,8 +12,6 @@ # CONFIG_SGI_IP22=y # CONFIG_SGI_IP27 is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_BOOT_ELF32=y CONFIG_ARC32=y CONFIG_BOARD_SCACHE=y
arch/mips64/defconfig-ip27 +0 -2
@@ -19,8 +19,6 @@ # CONFIG_REPLICATE_KTEXT is not set # CONFIG_REPLICATE_EXHANDLERS is not set CONFIG_SMP=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_BOOT_ELF64=y CONFIG_ARC64=y CONFIG_COHERENT_IO=y
arch/mips64/defconfig-ip32 +0 -2
@@ -13,8 +13,6 @@ # CONFIG_SGI_IP22 is not set # CONFIG_SGI_IP27 is not set CONFIG_SGI_IP32=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_BOOT_ELF32=y CONFIG_ARC32=y CONFIG_PC_KEYB=y
arch/parisc/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_PARISC y define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/ppc/config.in +0 -2
@@ -4,8 +4,6 @@ # see Documentation/kbuild/config-language.txt. # define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y mainmenu_name "Linux/PowerPC Kernel Configuration"
arch/ppc/configs/IVMS8_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/SM850_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/SPD823TS_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/TQM823L_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/TQM850L_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/TQM860L_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/apus_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/bseip_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/common_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/est8260_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/gemini_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/ibmchrp_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/mbx_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/oak_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/power3_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/rpxcllf_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/rpxlite_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/configs/walnut_defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/ppc/defconfig +0 -2
@@ -2,8 +2,6 @@ # Automatically generated make config: don't edit # # CONFIG_UID16 is not set -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # # Code maturity level options
arch/s390/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_EISA n define_bool CONFIG_MCA n define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/s390/defconfig +0 -2
@@ -5,8 +5,6 @@ # CONFIG_EISA is not set # CONFIG_MCA is not set CONFIG_UID16=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_ARCH_S390=y #
arch/s390x/config.in +0 -2
@@ -6,8 +6,6 @@ define_bool CONFIG_ISA n define_bool CONFIG_EISA n define_bool CONFIG_MCA n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/s390x/defconfig +0 -2
@@ -4,8 +4,6 @@ # CONFIG_ISA is not set # CONFIG_EISA is not set # CONFIG_MCA is not set -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_ARCH_S390=y CONFIG_ARCH_S390X=y
arch/sh/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_SUPERH y define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/sparc/config.in +0 -2
@@ -48,8 +48,6 @@ define_bool CONFIG_SUN_CONSOLE y define_bool CONFIG_SUN_AUXIO y define_bool CONFIG_SUN_IO y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc/defconfig +0 -2
@@ -38,8 +38,6 @@ CONFIG_SUN_CONSOLE=y CONFIG_SUN_AUXIO=y CONFIG_SUN_IO=y -CONFIG_RWSEM_GENERIC_SPINLOCK=y -# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_SUN4 is not set # CONFIG_PCI is not set CONFIG_SUN_OPENPROMFS=m
arch/sparc64/config.in +0 -2
@@ -33,8 +33,6 @@ # Global things across all Sun machines. define_bool CONFIG_HAVE_DEC_LOCK y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y define_bool CONFIG_ISA n define_bool CONFIG_ISAPNP n define_bool CONFIG_EISA n
arch/sparc64/defconfig +0 -2
@@ -23,8 +23,6 @@ CONFIG_SMP=y CONFIG_SPARC64=y CONFIG_HAVE_DEC_LOCK=y -# CONFIG_RWSEM_GENERIC_SPINLOCK is not set -CONFIG_RWSEM_XCHGADD_ALGORITHM=y # CONFIG_ISA is not set # CONFIG_ISAPNP is not set # CONFIG_EISA is not set
drivers/char/sysrq.c +3 -3
@@ -32,7 +32,7 @@#include <asm/ptrace.h> -extern void wakeup_bdflush (int ); +extern void wakeup_bdflush ();extern void reset_vc (unsigned int );extern struct list_head super_blocks;
@@ -221,7 +221,7 @@static void sysrq_handle_sync (int key,struct pt_regs *pt_regs,struct kbd_struct *kbd,struct tty_struct *tty) { emergency_sync_scheduled =EMERG_SYNC ; -wakeup_bdflush (0 ); +wakeup_bdflush (); }static struct sysrq_key_op sysrq_sync_op = { handler: sysrq_handle_sync,
@@ -232,7 +232,7 @@static void sysrq_handle_mountro (int key,struct pt_regs *pt_regs,struct kbd_struct *kbd,struct tty_struct *tty) { emergency_sync_scheduled =EMERG_REMOUNT ; -wakeup_bdflush (0 ); +wakeup_bdflush (); }static struct sysrq_key_op sysrq_mountro_op = { handler: sysrq_handle_mountro,
include/asm-alpha/rwsem.h +0 -208
208 lines not on this page · raw message
include/asm-i386/rwsem.h +0 -226
226 lines not on this page · raw message
include/asm-ppc/rwsem.h +0 -137
137 lines not on this page · raw message
include/asm-sparc64/rwsem.h +0 -233
233 lines not on this page · raw message
include/linux/rwsem-spinlock.h +0 -62
62 lines not on this page · raw message
include/linux/rwsem.h +55 -29
122 lines not on this page · raw message
lib/Makefile +2 -5
@@ -8,12 +8,9 @@ L_TARGET := lib.a -export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o +export-objs := cmdline.o dec_and_lock.o rwsem.o -obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o - -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o +obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o rwsem.o ifneq ($(CONFIG_HAVE_DEC_LOCK),y) obj-y += dec_and_lock.o
lib/rwsem-spinlock.c +0 -239
239 lines not on this page · raw message
lib/rwsem.c +38 -155
241 lines not on this page · raw message
Here's a patch to make rwsems unfair.
David
Quoted 2 lines · David Howells
Here's a patch to make rwsems unfair.
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even allows more than 2^31 concurrent readers on 64 bit archs ;).
just apply this patch on top of pre12:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre11aa1/00_rwsem-19
and then I'll send another incremental patch to kill the XCHGADD garbage. (but this patch will be enough to provide efficient and unfair rwsem to all architectures, so fixing the deadlock)
I inlined the patch below (may need some conversion to likely/unlikely as well).
arch/alpha/config.in +1 -2
@@ -5,8 +5,7 @@ define_bool CONFIG_ALPHA y define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y +define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y mainmenu_name "Kernel configuration of Linux for Alpha machines"
arch/arm/config.in +0 -2
@@ -10,8 +10,6 @@ define_bool CONFIG_MCA n define_bool CONFIG_UID16 y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n - mainmenu_option next_comment comment 'Code maturity level options'
arch/cris/config.in +0 -1
@@ -6,7 +6,6 @@ define_bool CONFIG_UID16 y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/i386/config.in +1 -3
@@ -51,7 +51,6 @@ define_bool CONFIG_X86_XADD n define_int CONFIG_X86_L1_CACHE_SHIFT 4 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n else define_bool CONFIG_X86_WP_WORKS_OK y define_bool CONFIG_X86_INVLPG y
@@ -59,8 +58,7 @@ define_bool CONFIG_X86_XADD y define_bool CONFIG_X86_BSWAP y define_bool CONFIG_X86_POPAD_OK y - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y + define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y fi if [ "$CONFIG_M486" = "y" ]; then define_int CONFIG_X86_L1_CACHE_SHIFT 4
arch/ia64/config.in +0 -1
@@ -24,7 +24,6 @@ define_bool CONFIG_MCA n define_bool CONFIG_SBUS n define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -1
@@ -5,7 +5,6 @@ define_bool CONFIG_UID16 y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux/68k Kernel Configuration"
arch/mips/config.in +0 -1
@@ -69,7 +69,6 @@ bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n # # Select some configuration options automatically for certain systems.
arch/mips64/config.in +0 -1
@@ -28,7 +28,6 @@ endmenu define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n # # Select some configuration options automatically based on user selections
arch/parisc/config.in +0 -1
@@ -8,7 +8,6 @@ define_bool CONFIG_PARISC y define_bool CONFIG_UID16 n define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/ppc/config.in +1 -2
@@ -4,8 +4,7 @@ # see Documentation/kbuild/config-language.txt. # define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y +define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y mainmenu_name "Linux/PowerPC Kernel Configuration"
arch/s390/config.in +0 -1
@@ -8,7 +8,6 @@ define_bool CONFIG_MCA n define_bool CONFIG_UID16 y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/s390x/config.in +0 -1
@@ -7,7 +7,6 @@ define_bool CONFIG_EISA n define_bool CONFIG_MCA n define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/sh/config.in +0 -1
@@ -8,7 +8,6 @@ define_bool CONFIG_UID16 y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/sparc/config.in +0 -1
@@ -49,7 +49,6 @@ define_bool CONFIG_SUN_AUXIO y define_bool CONFIG_SUN_IO y define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc64/config.in +2 -2
@@ -33,8 +33,8 @@ # Global things across all Sun machines. define_bool CONFIG_HAVE_DEC_LOCK y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y +# sorry I broke it again +define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y define_bool CONFIG_ISA n define_bool CONFIG_ISAPNP n define_bool CONFIG_EISA n
include/asm-alpha/rwsem_xchgadd.h +27 -0
@@ -0,0 +1,27 @@ +#ifndef _ALPHA_RWSEM_XCHGADD_H +#define _ALPHA_RWSEM_XCHGADD_H + +/* WRITEME */ + +static inline void __down_read (struct rw_semaphore *sem) +{ +} + +static inline void __down_write (struct rw_semaphore *sem) +{ +} + +static inline void __up_read (struct rw_semaphore *sem) +{ +} + +static inline void __up_write (struct rw_semaphore *sem) +{ +} + +static inline long rwsem_xchgadd (long value,long * count) +{ +return value; +} + +#endif
include/asm-i386/rwsem.h +0 -226
226 lines not on this page · raw message
include/asm-i386/rwsem_xchgadd.h +93 -0
93 lines not on this page · raw message
include/linux/rwsem-spinlock.h +0 -62
62 lines not on this page · raw message
include/linux/rwsem.h +5 -66
85 lines not on this page · raw message
include/linux/rwsem_spinlock.h +62 -0
62 lines not on this page · raw message
include/linux/rwsem_xchgadd.h +104 -0
104 lines not on this page · raw message
include/linux/sched.h +1 -1
@@ -239,7 +239,7 @@ pgd: swapper_pg_dir, \ mm_users: ATOMIC_INIT(2), \ mm_count: ATOMIC_INIT(1), \ - mmap_sem: __RWSEM_INITIALIZER(name.mmap_sem), \ + mmap_sem: RWSEM_INITIALIZER(name.mmap_sem), \ page_table_lock: SPIN_LOCK_UNLOCKED, \ mmlist: LIST_HEAD_INIT(name.mmlist), \ }
lib/Makefile +3 -3
@@ -8,12 +8,12 @@ L_TARGET := lib.a -export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o +export-objs := cmdline.o dec_and_lock.o rwsem_spinlock.o rwsem_xchgadd.o obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o +obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem_spinlock.o +obj-$(CONFIG_RWSEM_XCHGADD) += rwsem_xchgadd.o ifneq ($(CONFIG_HAVE_DEC_LOCK),y) obj-y += dec_and_lock.o
lib/rwsem.c +0 -210
210 lines not on this page · raw message
lib/rwsem_spinlock.c +126 -0
126 lines not on this page · raw message
lib/rwsem_xchgadd.c +92 -0
92 lines not on this page · raw message
Andrea
Quoted 2 lines · Andrea Arcangeli
I inlined the patch below (may need some conversion to likely/unlikely as well).
I don't know about you, but I'm mildly concerned that copyright attributions vanished.
-ben
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one
Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.
Quoted 1 line
(it even allows more than 2^31 concurrent readers on 64 bit archs ;).
Easy enough to fix. Just apply this as well:
include/linux/rwsem.h Wed Sep 19 19:23:47 2001 renamed from include/linux/rwsem.h Wed Sep 19 19:23:44 2001 +1 -1
@@ -26,7 +26,7 @@ * - if wait_list is not empty, then there are processes waiting for the semaphore */ struct rw_semaphore { - int activity; + long activity; spinlock_t lock; struct list_head wait_list; };
David
Quoted 3 lines
I don't know about you, but I'm mildly concerned that copyright attributions vanished.
I concur with that.
David
Quoted 2 lines · Benjamin LaHaise
I don't know about you, but I'm mildly concerned that copyright attributions vanished.
can you be more precise? which copyright attribution? my rwsem are written totally from scratch, they doesn't share anything with the previous code, it is just a code replacement. As far I can tell deleted some file, and with it also its headers but that seems not to infinge any copyright, I didn't removed only the copyright attribution. If I removed only the copyright attribution please let me know of course, that would been a silly mistake.
To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from rwsem_wake and friends to avoid saving it in the asm slow path, and I written that:
/*
* We return the semaphore itself from the C functions so we can pass it
* in %eax via regparm and we don't need to declare %eax clobbered by C.
* This is mostly for x86 but maybe other archs can make a use of it
* too.
* Idea is from David Howells <[email protected]>.
*/
And the xadd version is scheduled for removal anyways soon btw (David just dropped it in its implementation).
Andrea
Quoted 4 lines · David Howells
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one
Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.
then why your microbenchmarks says my version is much faster?
Andrea
Quoted 5 lines · David Howells
Quoted 3 lines
I don't know about you, but I'm mildly concerned that copyright attributions vanished.
I concur with that.
can you be a little more accurate on which copyright attributions vanished by mistake?
Andrea
Quoted 5 lines · Andrea Arcangeli
To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from
rwsem_wakeand friends to avoid saving it in the asm slow path, and I written that:
Your patch moved a bunch of code into asm-i386/rwsem_xchgadd.h. That code was derived from the spinlock code by me into the first rwsems, then David reworked bits of it, as wel as you. But there is no copyright on that file indicating this heritage. If you look at how strict commercial copyright control can be, even copying a single line of code mentally by retyping it can still mandate the copyright legacy. I'm sure it's just an oversight, but it's probably one we *all* need to be reminded of every now and again.
-ben
Quoted 7 lines · Benjamin LaHaise
Quoted 5 lines · Andrea Arcangeli
To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from
rwsem_wakeand friends to avoid saving it in the asm slow path, and I written that:
Quoted 8 lines
Your patch moved a bunch of code into
asm-i386/rwsem_xchgadd.h. That code was derived from the spinlock code by me into the first rwsems, then David reworked bits of it, as wel as you. But there is no copyright on that file indicating this heritage. If you look at how strict commercial copyright control can be, even copying a single line of code mentally by retyping it can still mandate the copyright legacy. I'm sure it's just an oversight, but it's probably one we *all* need to be reminded of every now and again.
I recall I wrote such code without copying anything, I certainly copied the way of doing things (pushl dx,cx and save ax via returning it in the slow path) as I wrote in the comment, but not the code itself. Infact at first i was probably also pushing eax, and even now that I use the same logic the two versions should be slightly different. and I think mine has a race this is why I'm not using such code and with the unfair thing such code is scheduled for removal since it's unfixable. really it sounds like too me too I cut and pasted the static inline void __down_read(struct rw_semaphore *sem) declarations, since I tend to left a space between "*" and the variable name but not always.
Andrea
PATCH 1/1 attempt #2 (Re: Deadlock on the mm->mmap_sem)
In reply to David HowellsHere's a revised patch to make rwsems unfair. This makes the activity counter a long (so it can handle up to 2^63 readers on a 64-bit arch) and fixes a small bug in the lib/rwsem.c file. This now passes all the rwsem tests that I've got.
David
Quoted 7 lines · Andrea Arcangeli
Quoted 4 lines · David Howells
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one
Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.
then why your microbenchmarks says my version is much faster?
They don't:
[my rwsem-unfair-2.diff patch]
#rds #wrs time : rd granted [spread] wr granted [spread] ==== ==== ===== =================== =================== 1 0 10s: 25384639 [0.008%] 0 [0.000%] 2 0 10s: 8007388 [0.013%] 0 [0.000%] 4 0 10s: 8004833 [0.007%] 0 [0.000%] 0 1 10s: 0 [0.000%] 25717288 [0.009%] 0 4 10s: 0 [0.000%] 2817293 [2.323%] 4 2 10s: 12742523 [0.008%] 114 [5.394%] 30 1 10s: 12739163 [0.002%] 92 [1.739%] 30 15 50s: 63733591 [0.003%] 286 [2.711%]
[Andrea's 00_rwsem-19 patch]
#rds #wrs time : rd granted [spread] wr granted [spread] ==== ==== ===== =================== =================== 1 0 10s: 24577938 [0.003%] 0 [0.000%] 2 0 10s: 5631224 [0.016%] 0 [0.000%] 4 0 10s: 5633280 [0.009%] 0 [0.000%] 0 1 10s: 0 [0.000%] 24572785 [0.007%] 0 4 10s: 0 [0.000%] 3444786 [0.091%] 4 2 10s: 8036291 [0.008%] 122 [4.918%] 30 1 10s: 8036490 [0.006%] 89 [1.253%] 30 15 50s: 40200472 [0.001%] 265 [2.717%]
[Unpatched kernel]
#rds #wrs time : rd granted [spread] wr granted [spread] ==== ==== ===== =================== =================== 1 0 10s: 30118903 [0.008%] 0 [0.000%] 2 0 10s: 12543737 [0.007%] 0 [0.000%] 4 0 10s: 12543731 [0.006%] 0 [0.000%] 0 1 10s: 0 [0.000%] 28408147 [0.008%] 0 4 10s: 0 [0.000%] 1554495 [0.172%] 4 2 10s: 1119656 [0.076%] 560020 [0.077%] 30 1 10s: 5361923 [0.024%] 55787 [0.012%] 30 15 50s: 5322473 [0.049%] 2661621 [0.050%]
David
Quoted 11 lines · David Howells
Quoted 7 lines · Andrea Arcangeli
Quoted 4 lines · David Howells
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one
Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.
then why your microbenchmarks says my version is much faster?
They don't:
ok, so I drop my objections, you must have changed something radical since the last time I benchmarked your generic rwsemaphores, I must have missed you did those changes, sorry, so I'll just drop my rwsem patch and I'll apply yours.
Andrea
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · David Howells
Quoted 4 lines · Andrea Arcangeli
Quoted 3 lines · David Howells
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.
then why your microbenchmarks says my version is much faster?
They don't:
ok, so I drop my objections, you must have changed something radical
hey no! It was a very unfair comparison! my rwsem are non inlined, yours are.
Andrea
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even
allows
Quoted 2 lines
more than 2^31 concurrent readers on 64 bit archs ;).
Andrea,
implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?
Multithreaded, iobound apps that use mmap are unusable with unfair semaphores.
My test app hangs for minutes within mprotect().
--
Manfred
Quoted 4 lines
I also don't think the hack is that bad. All it's doing is taking a copy of the process's VM decription so that it knows that nobody is going to modify it whilst a coredump is in progress.
You break the locking scheme of the mm structure.
Right now the rules are
1 get a mm_struct pointer by whatever means (walk the process list and
read task->mm, walk the mm_list)
2 increase mm_users
3 release the spinlock you acquired for 1
4 you can do with the result what you want.
With your patch applied, we would have to restrict rule 4 - at least modifying the vma list is not possible anymore, probably further changes.
AFAIK right now no external mm_struct user modifies the vma list, but it could be a problem in the future.
Quoted 6 lines
However, if you don't like that, how about just changing the lock on
mm_structto a special mm_struct-only type lock that has a recursive lock operation for use by the pagefault handler (and _only_ the pagefault handler)? I've attached a patch to do just that. This introduces five operations:
Does that solve the latency problem? That problem is pagefaults vs. another operation.
--
Manfred
Quoted 6 lines · Linus Torvalds
If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.
I've attached a recursive semaphore patch against 2.4.10-pre11 - but it makes mmap io unusable:
Testcase:
* file, mmaped, 300 MB, 128 MB Ram
* 2 threads: touch random pages
* third thread: calls mprotect(0xFFFF00000,0x1000, PAGE_READ)
Result:
mprotect hangs forever (minutes) with recursive semaphores.
With fair semaphores, mprotect returns after ~80 milliseconds with 5 worker threads, after ~380 milliseconds with 20 worker threads (slow IDE disk).
One alternative to David's patch would be moving the locking into the coredump handlers - would you prefer that?
--
Manfred
lib/rwsem-spinlock.c +1 -1
@@ -115,7 +115,7 @@spin_lock (&sem->wait_lock); -if (sem->activity>=0 &&list_empty (&sem->wait_list)) { +if (sem->activity>=0 ) {/* granted */ sem->activity++;spin_unlock (&sem->wait_lock);
arch/i386/config.in +2 -2
@@ -59,8 +59,8 @@ define_bool CONFIG_X86_XADD y define_bool CONFIG_X86_BSWAP y define_bool CONFIG_X86_POPAD_OK y - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y + define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y + define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n fi if [ "$CONFIG_M486" = "y" ]; then define_int CONFIG_X86_L1_CACHE_SHIFT 4
Quoted 9 lines · Manfred Spraul
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even
allows
Quoted 2 lines
more than 2^31 concurrent readers on 64 bit archs ;).
Andrea,
implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?
yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a mm_down_read macro. such way it will be recursive _and_ fair.
Andrea
Quoted 8 lines · Andrea Arcangeli
Quoted 7 lines · Manfred Spraul
Quoted 2 lines
if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even
allows
Quoted 2 lines
more than 2^31 concurrent readers on 64 bit archs ;).
Andrea,
Quoted 7 lines
Quoted 2 lines
implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?
yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a
mm_down_readmacro. such way it will be recursive _and_ fair.
ok, here it is attached a rwsem patch (now with the fast path inlined :) of my version of the rwsem-spinlock semaphores (as only option all over the ports). It's against pre12 (it is not inlined in the email since it's not readable anyways)
and below you find (this time inlined) an incremental patch to be applied on top of the attachment that implements the read recursive and at the same time fair rw semaphores. This should close all the problems.
But keep in mind the rwsem by default are _fair_, so you cannot do read recursion unless you use the recursive version and you pass a rw_sem_recursor to it.
arch/alpha/mm/fault.c +4 -4
@@ -113,7 +113,7 @@ goto vmalloc_fault;#endif -down_read (&mm->mmap_sem); +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -147,7 +147,7 @@* the fault. */ fault =handle_mm_fault (mm, vma, address, cause >0 ); -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);if (fault <0 ) goto out_of_memory;
@@ -161,7 +161,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);if (user_mode (regs)) {force_sig (SIGSEGV , current);
@@ -198,7 +198,7 @@if (current->pid ==1 ) { current->policy |=SCHED_YIELD ;schedule (); -down_read (&mm->mmap_sem); +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); goto survive; } printk(KERN_ALERT "VM: killing process %s(%d)\n",
arch/i386/mm/fault.c +6 -6
@@ -191,7 +191,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); vma =find_vma (mm, address);if (!vma)
@@ -265,7 +265,7 @@if (bit <32 ) tsk->thread.screen_bitmap |=1 << bit; } -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);return ;/*
@@ -273,7 +273,7 @@* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);/* User mode accesses just cause a SIGSEGV */ if (error_code &4 ) {
@@ -341,11 +341,11 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);if (tsk->pid ==1 ) { tsk->policy |=SCHED_YIELD ;schedule (); -down_read (&mm->mmap_sem); +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); goto survive; }printk ("VM: killing process %s\n" , tsk->comm);
@@ -354,7 +354,7 @@ goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);/* * Send a sigbus, regardless of whether we were in kernel
arch/ia64/mm/fault.c +4 -4
@@ -60,7 +60,7 @@if (in_interrupt () || !mm) goto no_context; -down_read (&mm->mmap_sem); +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); vma =find_vma_prev (mm, address, &prev_vma);if (!vma)
@@ -112,7 +112,7 @@default : goto out_of_memory; } -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);return ; check_expansion:
@@ -135,7 +135,7 @@ goto good_area; bad_area: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);if (isr &IA64_ISR_SP ) {/* * This fault was due to a speculative load set the "ed" bit in the psr to
@@ -184,7 +184,7 @@return ; out_of_memory: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);printk ("VM: killing process %s\n" , current->comm);if (user_mode (regs))do_exit (SIGKILL );
arch/ppc/mm/fault.c +5 -5
@@ -103,7 +103,7 @@bad_page_fault (regs, address,SIGSEGV );return ; } -down_read (&mm->mmap_sem); +down_read (&mm->mmap_sem, ¤t->mm_recursor); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -163,7 +163,7 @@ goto out_of_memory; } -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);/* * keep track of tlb+htab misses that are good addrs but * just need pte's created via handle_mm_fault()
@@ -173,7 +173,7 @@return ; bad_area: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); pte_errors++;/* User mode accesses cause a SIGSEGV */
@@ -194,7 +194,7 @@* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor);printk ("VM: killing process %s\n" , current->comm);if (user_mode (regs))do_exit (SIGKILL );
@@ -202,7 +202,7 @@return ; do_sigbus: -up_read (&mm->mmap_sem); +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); info.si_signo =SIGBUS ; info.si_errno =0 ; info.si_code =BUS_ADRERR ;
fs/exec.c +2 -2
@@ -969,9 +969,9 @@if (do_truncate (file->f_dentry,0 ) !=0 ) goto close_fail; -down_read (¤t->mm->mmap_sem); +down_read_recursive (¤t->mm->mmap_sem, ¤t->mm_recursor); retval = binfmt->core_dump (signr, regs, file); -up_read (¤t->mm->mmap_sem); +up_read_recursive (¤t->mm->mmap_sem, ¤t->mm_recursor); close_fail:filp_close (file,NULL );
fs/proc/array.c +8 -2
@@ -577,7 +577,10 @@ column = *ppos & (MAPS_LINE_LENGTH -1 );/* quickly go to line lineno */ -down_read (&mm->mmap_sem); +if (task == current) +down_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); +else +down_read (&mm->mmap_sem);for (map = mm->mmap, i =0 ; map && (i < lineno); map = map->vm_next, i++)continue ;
@@ -658,7 +661,10 @@if (volatile_task)break ; } -up_read (&mm->mmap_sem); +if (task == current) +up_read_recursive (&mm->mmap_sem, ¤t->mm_recursor); +else +up_read (&mm->mmap_sem);/* encode f_pos */ *ppos = (lineno <<MAPS_LINE_SHIFT ) + column;
include/linux/rwsem.h +34 -0
@@ -18,6 +18,11 @@#endif }; +struct rw_sem_recursor +{ +int counter; +}; +#if RWSEM_DEBUG #define __SEM_DEBUG_INIT (name) \ , (long)&(name).__magic
@@ -42,6 +47,7 @@ __SEM_DEBUG_INIT(name) \ }#define RWSEM_INITIALIZER (name) __RWSEM_INITIALIZER(name, 0) +#define RWSEM_RECURSOR_INITIALIZER ((struct rw_sem_recursor) { 0, })#define __DECLARE_RWSEM (name, count) \ struct rw_semaphore name = __RWSEM_INITIALIZER(name, count)
@@ -112,6 +118,34 @@spin_lock (&sem->lock); sem->count -=RWSEM_WRITE_BIAS ;if (unlikely (sem->count)) +rwsem_wake (sem); +spin_unlock (&sem->lock); +} + +static inline void down_read_recursive (struct rw_semaphore *sem, +struct rw_sem_recursor * recursor) +{ +int count, counter; +CHECK_MAGIC (sem->__magic); + +spin_lock (&sem->lock); + count = sem->count; + sem->count +=RWSEM_READ_BIAS ; + counter = recursor->counter++; +if (unlikely (count <0 && !counter && !(count &RWSEM_READ_MASK ))) +rwsem_down_failed (sem,RWSEM_READ_BLOCKING_BIAS ); +spin_unlock (&sem->lock); +} + +static inline void up_read_recursive (struct rw_semaphore *sem, +struct rw_sem_recursor * recursor) +{ +CHECK_MAGIC (sem->__magic); + +spin_lock (&sem->lock); + sem->count -=RWSEM_READ_BIAS ; + recursor->counter--; +if (unlikely (sem->count <0 && !(sem->count &RWSEM_READ_MASK )))rwsem_wake (sem);spin_unlock (&sem->lock); }
include/linux/sched.h +2 -0
@@ -315,6 +315,7 @@struct task_struct *next_task, *prev_task;struct mm_struct *active_mm; +struct rw_sem_recursor mm_recursor;struct list_head local_pages;unsigned int allocation_order, nr_local_pages;
@@ -460,6 +461,7 @@ policy: SCHED_OTHER, \ mm: NULL, \ active_mm: &init_mm, \ + mm_recursor: RWSEM_RECURSOR_INITIALIZER, \ cpus_allowed: -1, \ run_list: LIST_HEAD_INIT(tsk.run_list), \ next_task: &tsk, \
Andrea
arch/alpha/config.in +0 -2
@@ -5,8 +5,6 @@ define_bool CONFIG_ALPHA y define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y mainmenu_name "Kernel configuration of Linux for Alpha machines"
arch/arm/config.in +0 -2
@@ -9,8 +9,6 @@ define_bool CONFIG_SBUS n define_bool CONFIG_MCA n define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment
arch/cris/config.in +0 -2
@@ -5,8 +5,6 @@ mainmenu_name "Linux/CRIS Kernel Configuration" define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/i386/config.in +0 -4
@@ -50,8 +50,6 @@ define_bool CONFIG_X86_CMPXCHG n define_bool CONFIG_X86_XADD n define_int CONFIG_X86_L1_CACHE_SHIFT 4 - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n else define_bool CONFIG_X86_WP_WORKS_OK y define_bool CONFIG_X86_INVLPG y
@@ -59,8 +57,6 @@ define_bool CONFIG_X86_XADD y define_bool CONFIG_X86_BSWAP y define_bool CONFIG_X86_POPAD_OK y - define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n - define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y fi if [ "$CONFIG_M486" = "y" ]; then define_int CONFIG_X86_L1_CACHE_SHIFT 4
arch/ia64/config.in +0 -2
@@ -23,8 +23,6 @@ define_bool CONFIG_EISA n define_bool CONFIG_MCA n define_bool CONFIG_SBUS n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -2
@@ -4,8 +4,6 @@ # define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux/68k Kernel Configuration"
arch/mips/config.in +0 -2
@@ -68,8 +68,6 @@ fi bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000 -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n # # Select some configuration options automatically for certain systems.
arch/mips64/config.in +0 -3
@@ -27,9 +27,6 @@ fi endmenu -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n - # # Select some configuration options automatically based on user selections #
arch/parisc/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_PARISC y define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/ppc/config.in +0 -2
@@ -4,8 +4,6 @@ # see Documentation/kbuild/config-language.txt. # define_bool CONFIG_UID16 n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y mainmenu_name "Linux/PowerPC Kernel Configuration"
arch/s390/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_EISA n define_bool CONFIG_MCA n define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/s390x/config.in +0 -2
@@ -6,8 +6,6 @@ define_bool CONFIG_ISA n define_bool CONFIG_EISA n define_bool CONFIG_MCA n -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_name "Linux Kernel Configuration" define_bool CONFIG_ARCH_S390 y
arch/sh/config.in +0 -2
@@ -7,8 +7,6 @@ define_bool CONFIG_SUPERH y define_bool CONFIG_UID16 y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n mainmenu_option next_comment comment 'Code maturity level options'
arch/sparc/config.in +0 -2
@@ -48,8 +48,6 @@ define_bool CONFIG_SUN_CONSOLE y define_bool CONFIG_SUN_AUXIO y define_bool CONFIG_SUN_IO y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc64/config.in +0 -2
@@ -33,8 +33,6 @@ # Global things across all Sun machines. define_bool CONFIG_HAVE_DEC_LOCK y -define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n -define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y define_bool CONFIG_ISA n define_bool CONFIG_ISAPNP n define_bool CONFIG_EISA n
include/asm-alpha/rwsem.h +0 -208
208 lines not on this page · raw message
include/asm-i386/rwsem.h +0 -226
226 lines not on this page · raw message
include/linux/rwsem-spinlock.h +0 -62
62 lines not on this page · raw message
include/linux/rwsem.h +88 -48
168 lines not on this page · raw message
include/linux/sched.h +1 -1
@@ -239,7 +239,7 @@ pgd: swapper_pg_dir, \ mm_users: ATOMIC_INIT(2), \ mm_count: ATOMIC_INIT(1), \ - mmap_sem: __RWSEM_INITIALIZER(name.mmap_sem), \ + mmap_sem: RWSEM_INITIALIZER(name.mmap_sem), \ page_table_lock: SPIN_LOCK_UNLOCKED, \ mmlist: LIST_HEAD_INIT(name.mmlist), \ }
lib/Makefile +2 -5
@@ -8,12 +8,9 @@ L_TARGET := lib.a -export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o +export-objs := cmdline.o dec_and_lock.o rwsem.o -obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o - -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o +obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o rwsem.o ifneq ($(CONFIG_HAVE_DEC_LOCK),y) obj-y += dec_and_lock.o
lib/rwsem-spinlock.c +0 -239
239 lines not on this page · raw message
lib/rwsem.c +43 -190
253 lines not on this page · raw message
Quoted 4 lines · Andrea Arcangeli
yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a
mm_down_readmacro. such way it will be recursive _and_ fair.
You'd need a counter per-process per-mm_struct. Otherwise you couldn't do a recursive read lock simultaneously in two or more different processes, and also allow any one process to lock multiple mm_structs.
David
Quoted 10 lines · David Howells
Quoted 4 lines · Andrea Arcangeli
yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a
mm_down_readmacro. such way it will be recursive _and_ fair.You'd need a counter per-process per-mm_struct. Otherwise you couldn't do a recursive read lock simultaneously in two or more different processes, and also allow any one process to lock multiple
mm_structs.
the process doesn't need to lock multiple mm_structs at the same time.
I mean, we just need to allow a single task to go through, doesn't matter if the other tasks/threads are stuck, they will wait the write to finish. that's exactly where the fairness cames from.
The only thing that matters is that if a certain task passes the first read lock of its mmstruct semaphore, it will also pass any other further recursive lock again of its own _same_ mmstruct. So a per-process recursor is all what we need.
Must not be per-mm, per-mm would work but it would simply introduce the unfairness again.
Andrea
Quoted 1 line · Andrea Arcangeli
the process doesn't need to lock multiple
mm_structsat the same time.
fork, ptrace, /proc/pid/mem, /proc/pid/maps
All have to be able to lock two process's mm_structs simultaneously, even if it's indirectly through copy_to_user() or copy_from_user().
David
Quoted 8 lines · David Howells
Quoted 1 line · Andrea Arcangeli
the process doesn't need to lock multiple
mm_structsat the same time.fork, ptrace,
/proc/pid/mem,/proc/pid/mapsAll have to be able to lock two process's
mm_structssimultaneously, even if it's indirectly throughcopy_to_user()orcopy_from_user().
ptrace doesn't use down_read_recursive, nor /proc/<>/mem, nor fork.
for /proc/<pid>/maps this check takes care of it of course (or it could get unfair again: only when we're faulting on our vm we're allowed to go through):
if (task == current) down_read_recursive(&mm->mmap_sem, ¤t->mm_recursor); else down_read(&mm->mmap_sem);
Andrea