mlock: do not hold mmap_sem for extended periods of time
Subsystems (12)
cover mlock: do not hold mmap_sem for extended periods of time
Currently mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time, during which various commands such as 'ps auxw' will block. This makes sysadmins unhappy:
real 14m36.232s user 0m0.003s sys 0m0.015s
(output from 'time ps auxw' while a 20GB file was being mlocked without being previously preloaded into page cache)
I propose that mlock() could release mmap_sem after the VM_LOCKED bits have been set in all appropriate VMAs. Then a second pass could be done to actually mlock the pages, in small batches, releasing mmap_sem when we block on disk access or when we detect some contention.
Patches are against v2.6.37-rc4 plus my patches to avoid mlock dirtying (presently queued in -mm).
Michel Lespinasse (6):
mlock: only hold mmap_sem in shared mode when faulting in pages
mm: add FOLL_MLOCK follow_page flag.
mm: move VM_LOCKED check to __mlock_vma_pages_range()
rwsem: implement rwsem_is_contended()
mlock: do not hold mmap_sem for extended periods of time x86 rwsem: more precise rwsem_is_contended() implementation
arch/alpha/include/asm/rwsem.h | 5 + arch/ia64/include/asm/rwsem.h | 5 + arch/powerpc/include/asm/rwsem.h | 5 + arch/s390/include/asm/rwsem.h | 5 + arch/sh/include/asm/rwsem.h | 5 + arch/sparc/include/asm/rwsem.h | 5 + arch/x86/include/asm/rwsem.h | 35 ++++++--- arch/x86/lib/rwsem_64.S | 4 +- arch/x86/lib/semaphore_32.S | 4 +- arch/xtensa/include/asm/rwsem.h | 5 + include/linux/mm.h | 1 + include/linux/rwsem-spinlock.h | 1 + lib/rwsem-spinlock.c | 12 +++ mm/internal.h | 3 +- mm/memory.c | 54 ++++++++++++-- mm/mlock.c | 150 ++++++++++++++++++-------------------- mm/nommu.c | 6 +- 17 files changed, 201 insertions(+), 104 deletions(-)
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/6 mlock: only hold mmap_sem in shared mode when faulting in pages
Before this change, mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time. Various things will block while mmap_sem is held, including 'ps auxw'. This can make sysadmins angry.
I propose that mlock() could release mmap_sem after the VM_LOCKED bits have been set in all appropriate VMAs. Then a second pass could be done to actually mlock the pages with mmap_sem held for reads only. We need to recheck the vma flags after we re-acquire mmap_sem, but this is easy.
In the case where a vma has been munlocked before mlock completes, pages that were already marked as PageMlocked() are handled by the munlock() call, and mlock() is careful to not mark new page batches as PageMlocked() after the munlock() call has cleared the VM_LOCKED vma flags. So, the end result will be identical to what'd happen if munlock() had executed after the mlock() call.
In a later change, I will allow the second pass to release mmap_sem when blocking on disk accesses or when it is otherwise contended, so that it won't be held for long periods of time even in shared mode.
--- mm/mlock.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++------------- 1 files changed, 58 insertions(+), 17 deletions(-)
mm/mlock.c +58 -17
@@ -377,18 +377,10 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,int ret =0 ;int lock = newflags &VM_LOCKED ; -if (newflags == vma->vm_flags || - (vma->vm_flags & (VM_IO |VM_PFNMAP ))) +if (newflags == vma->vm_flags || (vma->vm_flags &VM_SPECIAL ) || +is_vm_hugetlb_page (vma) || vma ==get_gate_vma (current)) goto out;/* don't set VM_LOCKED, don't count */ -if ((vma->vm_flags & (VM_DONTEXPAND |VM_RESERVED )) || -is_vm_hugetlb_page (vma) || - vma ==get_gate_vma (current)) { -if (lock) -make_pages_present (start, end); - goto out;/* don't set VM_LOCKED, don't count */ - } - pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, newflags, vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma));
@@ -424,14 +416,10 @@ success:* set VM_LOCKED, __mlock_vma_pages_range will bring it back. */ -if (lock) { +if (lock) vma->vm_flags = newflags; - ret =__mlock_vma_pages_range (vma, start, end); -if (ret <0 ) - ret =__mlock_posix_error_return (ret); - }else { +else munlock_vma_pages_range (vma, start, end); - } out: *prev = vma;
@@ -444,7 +432,8 @@ static int do_mlock(unsigned long start, size_t len, int on)struct vm_area_struct * vma, * prev;int error; - len =PAGE_ALIGN (len); +VM_BUG_ON (start & ~PAGE_MASK ); +VM_BUG_ON (len !=PAGE_ALIGN (len)); end = start + len;if (end < start)return -EINVAL ;
@@ -487,6 +476,54 @@ static int do_mlock(unsigned long start, size_t len, int on)return error; } +static int do_mlock_pages (unsigned long start,size_t len) +{ +struct mm_struct *mm = current->mm; +unsigned long end, nstart, nend; +struct vm_area_struct *vma =NULL ; +int ret =0 ; + +VM_BUG_ON (start & ~PAGE_MASK ); +VM_BUG_ON (len !=PAGE_ALIGN (len)); + end = start + len; + +down_read (&mm->mmap_sem); +for (nstart = start; nstart < end; nstart = nend) { +/* +* We want to fault in pages for [nstart; end) address range. +* Find first corresponding VMA. +*/ +if (!vma) + vma =find_vma (mm, nstart); +else + vma = vma->vm_next; +if (!vma || vma->vm_start >= end) +break ; +/* +* Set [nstart; nend) to intersection of desired address +* range with the first VMA. Also, skip undesirable VMA types. +*/ + nend =min (end, vma->vm_end); +if (vma->vm_flags & (VM_IO |VM_PFNMAP )) +continue ; +if (nstart < vma->vm_start) + nstart = vma->vm_start; +/* +* Now fault in a range of pages within the first VMA. +*/ +if (vma->vm_flags &VM_LOCKED ) { + ret =__mlock_vma_pages_range (vma, nstart, nend); +if (ret) { + ret =__mlock_posix_error_return (ret); +break ; + } + }else +make_pages_present (nstart, nend); + } +up_read (&mm->mmap_sem); +return ret;/* 0 or negative error code */ +} + SYSCALL_DEFINE2(mlock, unsigned long, start, size_t, len) {unsigned long locked;
@@ -512,6 +549,8 @@ SYSCALL_DEFINE2(mlock, unsigned long, start, size_t, len)if ((locked <= lock_limit) ||capable (CAP_IPC_LOCK )) error =do_mlock (start, len,1 );up_write (¤t->mm->mmap_sem); +if (!error) + error =do_mlock_pages (start, len);return error; }
@@ -576,6 +615,8 @@ SYSCALL_DEFINE1(mlockall, int, flags) capable(CAP_IPC_LOCK)) ret =do_mlockall (flags);up_write (¤t->mm->mmap_sem); +if (!ret && (flags &MCL_CURRENT )) + ret =do_mlock_pages (0 ,TASK_SIZE ); out:return ret; }
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 10 lines
Currently
mlock()holdsmmap_semin exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time, during which various commands such as 'ps auxw' will block. This makes sysadmins unhappy:real 14m36.232s user 0m0.003s sys 0m0.015s(output from 'time ps auxw' while a 20GB file was being mlocked without being previously preloaded into page cache)
The kernel holds down_write(mmap_sem) for 14m36s?
geeze you guys are picky - that's less than a quarter hour!
Quoted 60 lines · On Thu, 2 Dec 2010 16:16:47 -0800 Michel Lespinasse <[email protected]> wrote:
Before this change,
mlock()holdsmmap_semin exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time. Various things will block whilemmap_semis held, including 'ps auxw'. This can make sysadmins angry.I propose that
mlock()could releasemmap_semafter theVM_LOCKEDbits have been set in all appropriate VMAs. Then a second pass could be done to actually mlock the pages withmmap_semheld for reads only. We need to recheck the vma flags after we re-acquiremmap_sem, but this is easy.In the case where a vma has been munlocked before mlock completes, pages that were already marked as
PageMlocked()are handled by themunlock()call, andmlock()is careful to not mark new page batches asPageMlocked()after themunlock()call has cleared theVM_LOCKEDvma flags. So, the end result will be identical to what'd happen ifmunlock()had executed after themlock()call.In a later change, I will allow the second pass to release
mmap_semwhen blocking on disk accesses or when it is otherwise contended, so that it won't be held for long periods of time even in shared mode....
mm/mlock.c +37
-0+static int do_mlock_pages (unsigned long start,size_t len) +{ +struct mm_struct *mm = current->mm; +unsigned long end, nstart, nend; +struct vm_area_struct *vma =NULL ; +int ret =0 ; + +VM_BUG_ON (start & ~PAGE_MASK ); +VM_BUG_ON (len !=PAGE_ALIGN (len)); + end = start + len; + +down_read (&mm->mmap_sem); +for (nstart = start; nstart < end; nstart = nend) { +/* +* We want to fault in pages for [nstart; end) address range. +* Find first corresponding VMA. +*/ +if (!vma) + vma =find_vma (mm, nstart); +else + vma = vma->vm_next; +if (!vma || vma->vm_start >= end) +break ; +/* +* Set [nstart; nend) to intersection of desired address +* range with the first VMA. Also, skip undesirable VMA types. +*/ + nend =min (end, vma->vm_end); +if (vma->vm_flags & (VM_IO |VM_PFNMAP )) +continue ; +if (nstart < vma->vm_start) + nstart = vma->vm_start; +/* +* Now fault in a range of pages within the first VMA. +*/ +if (vma->vm_flags &VM_LOCKED ) { + ret =__mlock_vma_pages_range (vma, nstart, nend);
mm/mlock.c static int do_mlock(unsigned long start, size_t len, int on) · line 525
Quoted 10 lines
mm/mlock.c +10
-0+if (ret) { + ret =__mlock_posix_error_return (ret); +break ; + } + }else +make_pages_present (nstart, nend); + } +up_read (&mm->mmap_sem); +return ret;/* 0 or negative error code */ +}
Am I correct in believing that we'll still hold down_read(mmap_sem) for a quarter hour?
If so, that's still pretty obnoxious behaviour - presumably there are workloads which will hurt like hell from that.
We don't need to hold mmap_sem at all while faulting in those pages, do we? We could just do
for (addr = start, addr < end; addr += PAGE_SIZE) get_user(x, addr);
and voila. If the pages are in cache and the ptes are set up then that will be *vastly* faster than the proposed code. If the get_user() takes a minor fault then it'll be slower. If it's a major fault then the difference probably doesn't matter much.
But whatever. Is this patchset a half-fix, and should we rather be looking for a full-fix?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 5 lines · Andrew Morton
Quoted 5 lines
Currently
mlock()holdsmmap_semin exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time, during which various commands such as 'ps auxw' will block. This makes sysadmins unhappy:
Quoted 7 lines
Quoted 5 lines
real 14m36.232s
user 0m0.003s
sys 0m0.015s
(output from 'time ps auxw' while a 20GB file was being mlocked without being previously preloaded into page cache)The kernel holds
down_write(mmap_sem)for 14m36s?
Yes...
[... patch snipped off ...]
Quoted 2 lines
Am I correct in believing that we'll still hold
down_read(mmap_sem)for a quarter hour?
Yes, patch 1/6 changes the long hold time to be in read mode instead of write mode, which is only a band-aid. But, this prepares for patch 5/6, which releases mmap_sem whenever there is contention on it or when blocking on disk reads.
Quoted 10 lines
We don't need to hold
mmap_semat all while faulting in those pages, do we? We could just dofor (addr = start, addr < end; addr +=
PAGE_SIZE)get_user(x, addr);and voila. If the pages are in cache and the ptes are set up then that will be *vastly* faster than the proposed code. If the
get_user()takes a minor fault then it'll be slower. If it's a major fault then the difference probably doesn't matter much.
get_user wouldn't suffice if the page is already mapped in, as we need to mark it as PageMlocked. Also, we need to skip IO and PFNMAP regions. I don't think you can make things much simpler than what I ended up with.
Quoted 2 lines
But whatever. Is this patchset a half-fix, and should we rather be looking for a full-fix?
I think the series fully fixes the mlock() and mlockall() cases, which has been the more pressing use case for us.
Even then, there are still cases where we could still observe long mmap_sem hold times - fundamentally, every place that calls get_user_pages (or do_mmap, in the mlockall MCL_FUTURE case) with a large page range may create such problems. From the looks of it, most of these places wouldn't actually care if the mmap_sem got dropped in the middle of the operation, but a general fix will have to involve looking at all the call sites to be sure.
Signature · 9 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 5 lines · Michel Lespinasse
Yes, patch 1/6 changes the long hold time to be in read mode instead of write mode, which is only a band-aid. But, this prepares for patch 5/6, which releases
mmap_semwhenever there is contention on it or when blocking on disk reads.
I have to say that I'm not a huge fan of that horribly kludgy contention check case.
The "move page-in to read-locked sequence" and the changes to get_user_pages look fine, but the contention thing is just disgusting. I'd really like to see some other approach if at all possible.
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Michel Lespinasse
Yes, patch 1/6 changes the long hold time to be in read mode instead of write mode, which is only a band-aid. But, this prepares for patch 5/6, which releases
mmap_semwhenever there is contention on it or when blocking on disk reads.
Quoted 6 lines
I have to say that I'm not a huge fan of that horribly kludgy contention check case.
The "move page-in to read-locked sequence" and the changes to
get_user_pageslook fine, but the contention thing is just disgusting. I'd really like to see some other approach if at all possible.
Are you OK with the part of patch 5/6 that drops mmap_sem when blocking on disk ?
This by itself brings mmap_sem hold time down to a few seconds. Plus, I could add something to limit the interval passed to __mlock_vma_pages_range to a thousand pages or so, so that the hold time would then be bounded by that constant.
I think rwsem_is_contended() actually sounds better than fiddling with constants, but OTOH maybe the mlock use case is not significant enough to justify introducing that new API.
Signature · 9 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 3 lines · Michel Lespinasse
I think
rwsem_is_contended()actually sounds better than fiddling with constants, but OTOH maybe the mlock use case is not significant enough to justify introducing that new API.
Right, so I don't see the problem with _is_contended() either. In fact, I introduce mutex_is_contended() in the mmu_preempt series to convert existing (spin) lock break tests.
If you want to do lock-breaks like cond_resched_lock() all you really have is *_is_contended(), sleeping locks will schedule unconditional.
int cond_break_mutex(struct mutex *mutex)
{
int ret = 0;
if (mutex_is_contended(mutex)) {
mutex_unlock(mutex);
ret = 1;
mutex_lock(mutex);
}
return 1;
}Or more exotic lock breaks, like the mmu-gather stuff, which falls out of its nested page-table loops and restarts the whole affair.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
In reply to Linus TorvaldsQuoted 7 lines · Linus Torvalds
Quoted 5 lines · Michel Lespinasse
Yes, patch 1/6 changes the long hold time to be in read mode instead of write mode, which is only a band-aid. But, this prepares for patch 5/6, which releases
mmap_semwhenever there is contention on it or when blocking on disk reads.
Quoted 6 lines
I have to say that I'm not a huge fan of that horribly kludgy contention check case.
The "move page-in to read-locked sequence" and the changes to
get_user_pageslook fine, but the contention thing is just disgusting. I'd really like to see some other approach if at all possible.
Andrew, should I amend my patches to remove the rwsem_is_contended() code ? This would involve:
- remove rwsem-implement-rwsem_is_contended.patch and x86-rwsem-more-precise-rwsem_is_contended-implementation.patch
- in mlock-do-not-hold-mmap_sem-for-extended-periods-of-time.patch, drop the one hunk making use of rwsem_is_contended (rest of the patch would still work without it)
- optionally, follow up patch to limit batch size to a constant in do_mlock_pages():
mm/mlock.c +8 -0
@@ -457,15 +457,23 @@ static int do_mlock_pages(unsigned long start, size_t len)continue ;if (nstart < vma->vm_start) nstart = vma->vm_start; +/* +* Constrain batch size to limit mmap_sem hold time. +*/ +if (nend > nstart +1024 *PAGE_SIZE ) + nend = nstart +1024 *PAGE_SIZE ;/* * Now fault in a range of pages. __mlock_vma_pages_range() * double checks the vma flags, so that it won't mlock pages * if the vma was already munlocked. */ ret =__mlock_vma_pages_range (vma, nstart, nend, &locked);if (ret <0 ) { ret =__mlock_posix_error_return (ret);break ; + }else if (locked) { + locked =0 ; +up_read (&mm->mmap_sem); } nend = nstart + ret *PAGE_SIZE ; ret =0 ;
I don't really prefer using a constant, but I'm not sure how else to make Linus happy :)
Signature · 9 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 15 lines · On Mon, 13 Dec 2010 16:51:40 -0800 Michel Lespinasse <[email protected]> wrote:
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Michel Lespinasse
Yes, patch 1/6 changes the long hold time to be in read mode instead of write mode, which is only a band-aid. But, this prepares for patch 5/6, which releases
mmap_semwhenever there is contention on it or when blocking on disk reads.Quoted 6 lines
I have to say that I'm not a huge fan of that horribly kludgy contention check case.
The "move page-in to read-locked sequence" and the changes to
get_user_pageslook fine, but the contention thing is just disgusting. I'd really like to see some other approach if at all possible.
Quoted 7 lines
Andrew, should I amend my patches to remove the
rwsem_is_contended()code ? This would involve:
- remove rwsem-implement-rwsem_is_contended.patch and x86-rwsem-more-precise-rwsem_is_contended-implementation.patch
- in mlock-do-not-hold-mmap_sem-for-extended-periods-of-time.patch, drop the one hunk making use ofrwsem_is_contended(rest of the patch would still work without it)
I think I fixed all that up.
Quoted 33 lines
- optionally, follow up patch to limit batch size to a constant in
do_mlock_pages():mm/mlock.c +8
-0@@ -457,15 +457,23 @@ static int do_mlock_pages(unsigned long start, size_t len)continue ;if (nstart < vma->vm_start) nstart = vma->vm_start; +/* +* Constrain batch size to limit mmap_sem hold time. +*/ +if (nend > nstart +1024 *PAGE_SIZE ) + nend = nstart +1024 *PAGE_SIZE ;/* * Now fault in a range of pages. __mlock_vma_pages_range() * double checks the vma flags, so that it won't mlock pages * if the vma was already munlocked. */ ret =__mlock_vma_pages_range (vma, nstart, nend, &locked);if (ret <0 ) { ret =__mlock_posix_error_return (ret);break ; + }else if (locked) { + locked =0 ; +up_read (&mm->mmap_sem); } nend = nstart + ret *PAGE_SIZE ; ret =0 ;
Quoted 2 lines
I don't really prefer using a constant, but I'm not sure how else to make Linus happy :)
rwsem_is_contended() didn't seem so bad to me.
Reading 1024 pages can still take a long time. I can't immediately think of a better approach though.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 11 lines · Andrew Morton
Quoted 7 lines · On Mon, 13 Dec 2010 16:51:40 -0800 Michel Lespinasse <[email protected]> wrote:
Andrew, should I amend my patches to remove the
rwsem_is_contended()code ? This would involve:
- remove rwsem-implement-rwsem_is_contended.patch and x86-rwsem-more-precise-rwsem_is_contended-implementation.patch
- in mlock-do-not-hold-mmap_sem-for-extended-periods-of-time.patch, drop the one hunk making use ofrwsem_is_contended(rest of the patch would still work without it)I think I fixed all that up.
Thanks!
Quoted 2 lines
Quoted 2 lines
- optionally, follow up patch to limit batch size to a constant in
do_mlock_pages():
[... diff snipped ...]
Quoted 7 lines
Quoted 2 lines
I don't really prefer using a constant, but I'm not sure how else to make Linus happy :)
rwsem_is_contended()didn't seem so bad to me.Reading 1024 pages can still take a long time. I can't immediately think of a better approach though.
Note that we're only concerned page cache hist here, as __get_user_pages with non-NULL nonblocking argument will release mmap_sem when blocking on disk. So time per page is somewhat constant
- we don't need to worry about disk seeks at least.
Signature · 9 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
In reply to Andrew MortonQuoted 3 lines · Andrew Morton
Reading 1024 pages can still take a long time. I can't immediately think of a better approach though.
I don't see the need for _any_ of this.
Guys, we used to hold the damn thing for writing the *WHOLE*DAMN*TIME*.
Without _any_ at all of the crappy "rwsem_contended()" or the stupid constants, we hold it only for reading, _and_ we drop it for any actual IO. So the semaphore is held only for actual CPU intensive cases. We're talking a reduction from minutes to milliseconds.
So stop this insanity. Do neither the rwsem contention checking _nor_ the "do things in batches".
Really.
The thing is, afte six months of doing the simple and straightforward and _obvious_ parts, if people still think it's a real problem, at that point I'm going to be interested in hearing about trying to be clever. But when the semaphore hold times have gone down by four orders of magnitude, I simply think it's fundamentally wrong to dick around with some stupid detail. Certainly not in the same patch series.
"Keep It Simple, Stupid".
So don't even _try_ to send me a series that does all of this. I'm not going to take it. Do a series that fixes the _problem_. No more.
And btw, read the paper "Worse is better".
Linus
Re: [PATCH 1/6] mlock: only hold mmap_sem in shared mode when faulting in pages
Quoted 4 lines · Linus Torvalds
Without _any_ at all of the crappy "
rwsem_contended()" or the stupid constants, we hold it only for reading, _and_ we drop it for any actual IO. So the semaphore is held only for actual CPU intensive cases. We're talking a reduction from minutes to milliseconds.
It's actually still several seconds for a large enough mlock from page cache.
But yes, I agree it'll do fine for now :)
Signature · 9 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/6 mm: add FOLL_MLOCK follow_page flag.
In reply to Michel LespinasseMove the code to mlock pages from __mlock_vma_pages_range() to follow_page().
This allows __mlock_vma_pages_range() to not have to break down work into 16-page batches.
An additional motivation for doing this within the present patch series is that it'll make it easier for a later chagne to drop mmap_sem when blocking on disk (we'd like to be able to resume at the page that was read from disk instead of at the start of a 16-page batch).
--- include/linux/mm.h | 1 + mm/memory.c | 27 ++++++++++++++++++++- mm/mlock.c | 65 ++++------------------------------------------------ 3 files changed, 32 insertions(+), 61 deletions(-)
include/linux/mm.h +1 -0
@@ -1415,6 +1415,7 @@ struct page *follow_page(struct vm_area_struct *, unsigned long address,#define FOLL_GET 0x04/* do get_page on page */ #define FOLL_DUMP 0x08/* give error on hole if it would be zero */ #define FOLL_FORCE 0x10/* get_user_pages read/write w/o permission */ +#define FOLL_MLOCK 0x20/* mark page as mlocked */ typedef int (*pte_fn_t )(pte_t *pte,pgtable_t token,unsigned long addr,void *data);
mm/memory.c +26 -1
@@ -1297,7 +1297,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, page =pte_page (pte); } -if (flags &FOLL_GET ) +if (flags & (FOLL_GET |FOLL_MLOCK ))get_page (page);if (flags &FOLL_TOUCH ) { if ((flags & FOLL_WRITE) &&
@@ -1310,6 +1310,31 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,*/ mark_page_accessed (page); } +if (flags &FOLL_MLOCK ) { +pte_unmap_unlock (ptep, ptl); +lru_add_drain ();/* push cached pages to LRU */ +if (page->mapping) { +/* +* That preliminary check is mainly to avoid +* the pointless overhead of lock_page on the +* ZERO_PAGE: which might bounce very badly if +* there is contention. However, we're still +* dirtying its cacheline with get/put_page. +*/ +lock_page (page); +/* +* Because we lock page here and migration is +* blocked by the elevated reference, we need +* only check for file-cache page truncation. +*/ +if (page->mapping) +mlock_vma_page (page); +unlock_page (page); + } +VM_BUG_ON (flags &FOLL_GET ); +put_page (page); +return page; + } unlock:pte_unmap_unlock (ptep, ptl); out:
mm/mlock.c +5 -60
85 lines not on this page · raw message
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/6] mm: add FOLL_MLOCK follow_page flag.
I did not realize that previously, but it turns out to be possible to manipulate LRU lists under control of the pte lock, rather than having to get/put the page. This allows the FOLL_MLOCK follow_page flag implementation to look much nicer than it did in initial proposal.
Please consider the following as a replacement for the initial patch 2/6 proposal...
--------------------------------- 8< --------------------------------------
Move the code to mlock pages from __mlock_vma_pages_range() to follow_page().
This allows __mlock_vma_pages_range() to not have to break down work into 16-page batches.
An additional motivation for doing this within the present patch series is that it'll make it easier for a later chagne to drop mmap_sem when blocking on disk (we'd like to be able to resume at the page that was read from disk instead of at the start of a 16-page batch).
--- include/linux/mm.h | 1 + mm/memory.c | 22 +++++++++++++++++ mm/mlock.c | 65 ++++------------------------------------------------ 3 files changed, 28 insertions(+), 60 deletions(-)
include/linux/mm.h +1 -0
@@ -1415,6 +1415,7 @@ struct page *follow_page(struct vm_area_struct *, unsigned long address,#define FOLL_GET 0x04/* do get_page on page */ #define FOLL_DUMP 0x08/* give error on hole if it would be zero */ #define FOLL_FORCE 0x10/* get_user_pages read/write w/o permission */ +#define FOLL_MLOCK 0x20/* mark page as mlocked */ typedef int (*pte_fn_t )(pte_t *pte,pgtable_t token,unsigned long addr,void *data);
mm/memory.c +22 -0
@@ -1310,6 +1310,28 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,*/ mark_page_accessed (page); } +if (flags &FOLL_MLOCK ) { +/* +* The preliminary mapping check is mainly to avoid the +* pointless overhead of lock_page on the ZERO_PAGE +* which might bounce very badly if there is contention. +* +* If the page is already locked, we don't need to +* handle it now - vmscan will handle it later if and +* when it attempts to reclaim the page. +*/ +if (page->mapping &&trylock_page (page)) { +lru_add_drain ();/* push cached pages to LRU */ +/* +* Because we lock page here and migration is +* blocked by the pte's page reference, we need +* only check for file-cache page truncation. +*/ +if (page->mapping) +mlock_vma_page (page); +unlock_page (page); + } + } unlock:pte_unmap_unlock (ptep, ptl); out:
mm/mlock.c +5 -60
85 lines not on this page · raw message
Signature · 7 lines
1.7.3.1 -- Michel "Walken" Lespinasse A program is never fully debugged until the last user dies.
PATCH 3/6 mm: move VM_LOCKED check to __mlock_vma_pages_range()
In reply to Michel LespinasseUse a single code path for faulting in pages during mlock.
The reason to have it in this patch series is that I did not want to update both code paths in a later change that releases mmap_sem when blocking on disk.
--- mm/mlock.c | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-)
mm/mlock.c +9 -9
@@ -169,7 +169,7 @@ static long __mlock_vma_pages_range(struct vm_area_struct *vma,VM_BUG_ON (end > vma->vm_end);VM_BUG_ON (!rwsem_is_locked (&mm->mmap_sem)); - gup_flags =FOLL_TOUCH |FOLL_MLOCK ; + gup_flags =FOLL_TOUCH ;/* * We want to touch writable mappings with a write fault in order * to break COW, except for shared mappings because these don't COW
@@ -178,6 +178,9 @@ static long __mlock_vma_pages_range(struct vm_area_struct *vma,if ((vma->vm_flags & (VM_WRITE |VM_SHARED )) ==VM_WRITE ) gup_flags |=FOLL_WRITE ; +if (vma->vm_flags &VM_LOCKED ) + gup_flags |=FOLL_MLOCK ; +/* We don't try to access the guard page of a stack vma */ if (stack_guard_page (vma, start)) { addr +=PAGE_SIZE ;
@@ -456,14 +459,11 @@ static int do_mlock_pages(unsigned long start, size_t len)/* * Now fault in a range of pages within the first VMA. */ -if (vma->vm_flags &VM_LOCKED ) { - ret =__mlock_vma_pages_range (vma, nstart, nend); -if (ret) { - ret =__mlock_posix_error_return (ret); -break ; - } - }else -make_pages_present (nstart, nend); + ret =__mlock_vma_pages_range (vma, nstart, nend); +if (ret) { + ret =__mlock_posix_error_return (ret); +break ; + } }up_read (&mm->mmap_sem);return ret;/* 0 or negative error code */
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/6 rwsem: implement rwsem_is_contended()
In reply to Michel LespinasseTrivial implementations for rwsem_is_contended()
--- arch/alpha/include/asm/rwsem.h | 5 +++++ arch/ia64/include/asm/rwsem.h | 5 +++++ arch/powerpc/include/asm/rwsem.h | 5 +++++ arch/s390/include/asm/rwsem.h | 5 +++++ arch/sh/include/asm/rwsem.h | 5 +++++ arch/sparc/include/asm/rwsem.h | 5 +++++ arch/x86/include/asm/rwsem.h | 5 +++++ arch/xtensa/include/asm/rwsem.h | 5 +++++ include/linux/rwsem-spinlock.h | 1 + lib/rwsem-spinlock.c | 12 ++++++++++++ 10 files changed, 53 insertions(+), 0 deletions(-)
arch/alpha/include/asm/rwsem.h +5 -0
@@ -255,5 +255,10 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return (sem->count !=0 ); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* __KERNEL__ */ #endif /* _ALPHA_RWSEM_H */
arch/ia64/include/asm/rwsem.h +5 -0
@@ -179,4 +179,9 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem) return (sem->count != 0); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* _ASM_IA64_RWSEM_H */
arch/powerpc/include/asm/rwsem.h +5 -0
@@ -179,5 +179,10 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return sem->count !=0 ; } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return sem->count <0 ; +} +#endif /* __KERNEL__ */ #endif /* _ASM_POWERPC_RWSEM_H */
arch/s390/include/asm/rwsem.h +5 -0
@@ -382,5 +382,10 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return (sem->count !=0 ); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* __KERNEL__ */ #endif /* _S390_RWSEM_H */
arch/sh/include/asm/rwsem.h +5 -0
@@ -184,5 +184,10 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return (sem->count !=0 ); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* __KERNEL__ */ #endif /* _ASM_SH_RWSEM_H */
arch/sparc/include/asm/rwsem.h +5 -0
@@ -165,6 +165,11 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return (sem->count !=0 ); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* __KERNEL__ */ #endif /* _SPARC64_RWSEM_H */
arch/x86/include/asm/rwsem.h +5 -0
@@ -275,5 +275,10 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)return (sem->count !=0 ); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* __KERNEL__ */ #endif /* _ASM_X86_RWSEM_H */
arch/xtensa/include/asm/rwsem.h +5 -0
@@ -165,4 +165,9 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem) return (sem->count != 0); } +static inline int rwsem_is_contended (struct rw_semaphore *sem) +{ +return (sem->count <0 ); +} +#endif /* _XTENSA_RWSEM_H */
include/linux/rwsem-spinlock.h +1 -0
@@ -69,6 +69,7 @@ extern void __up_read(struct rw_semaphore *sem);extern void __up_write (struct rw_semaphore *sem);extern void __downgrade_write (struct rw_semaphore *sem);extern int rwsem_is_locked (struct rw_semaphore *sem); +extern int rwsem_is_contended (struct rw_semaphore *sem);#endif /* __KERNEL__ */ #endif /* _LINUX_RWSEM_SPINLOCK_H */
lib/rwsem-spinlock.c +12 -0
@@ -30,6 +30,18 @@ int rwsem_is_locked(struct rw_semaphore *sem) }EXPORT_SYMBOL (rwsem_is_locked); +int rwsem_is_contended (struct rw_semaphore *sem) +{ +int ret =0 ; +unsigned long flags; + +if (spin_trylock_irqsave (&sem->wait_lock, flags)) { + ret = !list_empty (&sem->wait_list); +spin_unlock_irqrestore (&sem->wait_lock, flags); + } +return ret; +} +/* * initialise the semaphore */
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/6 mlock: do not hold mmap_sem for extended periods of time
In reply to Michel Lespinasse__get_user_pages gets a new 'nonblocking' parameter to signal that the caller is prepared to re-acquire mmap_sem and retry the operation if needed. This is used to split off long operations if they are going to block on a disk transfer, or when we detect contention on the mmap_sem.
--- mm/internal.h | 3 ++- mm/memory.c | 27 ++++++++++++++++++++++----- mm/mlock.c | 34 ++++++++++++++++++++-------------- mm/nommu.c | 6 ++++-- 4 files changed, 48 insertions(+), 22 deletions(-)
mm/internal.h +2 -1
@@ -243,7 +243,8 @@ static inline void mminit_validate_memmodel_limits(unsigned long *start_pfn,int __get_user_pages (struct task_struct *tsk,struct mm_struct *mm,unsigned long start,int len,unsigned int foll_flags, -struct page **pages,struct vm_area_struct **vmas); +struct page **pages,struct vm_area_struct **vmas, +int *nonblocking);#define ZONE_RECLAIM_NOSCAN -2#define ZONE_RECLAIM_FULL -1
mm/memory.c +22 -5
@@ -1366,7 +1366,8 @@ no_page_table:int __get_user_pages (struct task_struct *tsk,struct mm_struct *mm,unsigned long start,int nr_pages,unsigned int gup_flags, -struct page **pages,struct vm_area_struct **vmas) +struct page **pages,struct vm_area_struct **vmas, +int *nonblocking) {int i;unsigned long vm_flags;
@@ -1465,11 +1466,15 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,cond_resched ();while (!(page =follow_page (vma, start, foll_flags))) { +int fault_flags =0 ;int ret; +if (foll_flags &FOLL_WRITE ) + fault_flags |=FAULT_FLAG_WRITE ; +if (nonblocking) + fault_flags |=FAULT_FLAG_ALLOW_RETRY ; ret =handle_mm_fault (mm, vma, start, - (foll_flags &FOLL_WRITE ) ? -FAULT_FLAG_WRITE :0 ); + fault_flags);if (ret &VM_FAULT_ERROR ) {if (ret &VM_FAULT_OOM )
@@ -1485,6 +1490,11 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,else tsk->min_flt++; +if (ret &VM_FAULT_RETRY ) { + *nonblocking =0 ; +return i; + } +/* * The VM_FAULT_WRITE bit tells us that * do_wp_page has broken COW when necessary,
@@ -1516,6 +1526,11 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm, i++; start +=PAGE_SIZE ; nr_pages--; +if (nonblocking &&rwsem_is_contended (&mm->mmap_sem)) { +up_read (&mm->mmap_sem); + *nonblocking =0 ; +return i; + } }while (nr_pages && start < vma->vm_end); }while (nr_pages);return i;
@@ -1584,7 +1599,8 @@ int get_user_pages(struct task_struct *tsk, struct mm_struct *mm,if (force) flags |=FOLL_FORCE ; -return __get_user_pages (tsk, mm, start, nr_pages, flags, pages, vmas); +return __get_user_pages (tsk, mm, start, nr_pages, flags, pages, vmas, +NULL ); }EXPORT_SYMBOL (get_user_pages);
@@ -1609,7 +1625,8 @@ struct page *get_dump_page(unsigned long addr)struct page *page;if (__get_user_pages (current, current->mm, addr,1 , -FOLL_FORCE |FOLL_DUMP |FOLL_GET , &page, &vma) <1 ) +FOLL_FORCE |FOLL_DUMP |FOLL_GET , &page, &vma, +NULL ) <1 )return NULL ;flush_cache_page (vma, addr,page_to_pfn (page));return page;
mm/mlock.c +19 -13
@@ -155,13 +155,13 @@ static inline int stack_guard_page(struct vm_area_struct *vma, unsigned long add* vma->vm_mm->mmap_sem must be held for at least read. */ static long __mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end) +unsigned long start,unsigned long end, +int *nonblocking) {struct mm_struct *mm = vma->vm_mm;unsigned long addr = start;int nr_pages = (end - start) /PAGE_SIZE ;int gup_flags; -int ret;VM_BUG_ON (start & ~PAGE_MASK );VM_BUG_ON (end & ~PAGE_MASK );
@@ -187,9 +187,8 @@ static long __mlock_vma_pages_range(struct vm_area_struct *vma, nr_pages--; } - ret =__get_user_pages (current, mm, addr, nr_pages, gup_flags, -NULL ,NULL ); -return max (ret,0 );/* 0 or negative error code */ +return __get_user_pages (current, mm, addr, nr_pages, gup_flags, +NULL ,NULL , nonblocking); }/*
@@ -233,7 +232,7 @@ long mlock_vma_pages_range(struct vm_area_struct *vma, is_vm_hugetlb_page(vma) || vma == get_gate_vma(current))) { -__mlock_vma_pages_range (vma, start, end); +__mlock_vma_pages_range (vma, start, end,NULL );/* Hide errors from mmap() and other callers */ return 0 ;
@@ -429,21 +428,23 @@ static int do_mlock_pages(unsigned long start, size_t len)struct mm_struct *mm = current->mm;unsigned long end, nstart, nend;struct vm_area_struct *vma =NULL ; +int locked =0 ;int ret =0 ;VM_BUG_ON (start & ~PAGE_MASK );VM_BUG_ON (len !=PAGE_ALIGN (len)); end = start + len; -down_read (&mm->mmap_sem);for (nstart = start; nstart < end; nstart = nend) {/* * We want to fault in pages for [nstart; end) address range. * Find first corresponding VMA. */ -if (!vma) +if (!locked) { + locked =1 ; +down_read (&mm->mmap_sem); vma =find_vma (mm, nstart); -else + }else if (nstart >= vma->vm_end) vma = vma->vm_next;if (!vma || vma->vm_start >= end)break ;
@@ -456,16 +457,21 @@ static int do_mlock_pages(unsigned long start, size_t len)continue ;if (nstart < vma->vm_start) nstart = vma->vm_start;/* -* Now fault in a range of pages within the first VMA. +* Now fault in a range of pages. __mlock_vma_pages_range() +* double checks the vma flags, so that it won't mlock pages +* if the vma was already munlocked. */ - ret =__mlock_vma_pages_range (vma, nstart, nend); -if (ret) { + ret =__mlock_vma_pages_range (vma, nstart, nend, &locked); +if (ret <0 ) { ret =__mlock_posix_error_return (ret);break ; } + nend = nstart + ret *PAGE_SIZE ; + ret =0 ; } -up_read (&mm->mmap_sem); +if (locked) +up_read (&mm->mmap_sem);return ret;/* 0 or negative error code */ }
mm/nommu.c +4 -2
@@ -127,7 +127,8 @@ unsigned int kobjsize(const void *objp)int __get_user_pages (struct task_struct *tsk,struct mm_struct *mm,unsigned long start,int nr_pages,unsigned int foll_flags, -struct page **pages,struct vm_area_struct **vmas) +struct page **pages,struct vm_area_struct **vmas, +int *retry) {struct vm_area_struct *vma;unsigned long vm_flags;
@@ -185,7 +186,8 @@ int get_user_pages(struct task_struct *tsk, struct mm_struct *mm,if (force) flags |=FOLL_FORCE ; -return __get_user_pages (tsk, mm, start, nr_pages, flags, pages, vmas); +return __get_user_pages (tsk, mm, start, nr_pages, flags, pages, vmas, +NULL ); }EXPORT_SYMBOL (get_user_pages);
Signature · 1 line
1.7.3.1
Andrew Morton
maintainerQuoted 4 lines · On Thu, 2 Dec 2010 16:16:51 -0800 Michel Lespinasse <[email protected]> wrote:
__get_user_pagesgets a new 'nonblocking' parameter to signal that the caller is prepared to re-acquiremmap_semand retry the operation if needed. This is used to split off long operations if they are going to block on a disk transfer, or when we detect contention on themmap_sem.
Doesn't apply to linux-next because the KVM guys snuck in a new FAULT_FLAG_MINOR (who knew?). With a bonus, undocumented, exported-to-modules get_user_pages_noio().
I liked your code better so I munged __get_user_pages() together thusly:
cond_resched();
while (!(page = follow_page(vma, start, foll_flags))) {
int ret;
unsigned int fault_flags = 0;
if (foll_flags & FOLL_WRITE)
fault_flags |= FAULT_FLAG_WRITE;
if (nonblocking)
fault_flags |= FAULT_FLAG_ALLOW_RETRY;
if (foll_flags & FOLL_MINOR)
fault_flags |= FAULT_FLAG_MINOR;
ret = handle_mm_fault(mm, vma, start,
fault_flags);please review the end result..
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/6 x86 rwsem: more precise rwsem_is_contended() implementation
In reply to Michel LespinasseWe would like rwsem_is_contended() to return true only once a contending writer has had a chance to insert itself onto the rwsem wait queue. To that end, we need to differenciate between active and queued writers.
A new property is introduced: RWSEM_ACTIVE_WRITE_BIAS is set to be 'more negative' than RWSEM_WAITING_BIAS. RWSEM_WAITING_MASK designates a bit in the rwsem count that will be set only when RWSEM_WAITING_BIAS is in effect.
The basic properties that have been true so far also still hold:
- RWSEM_ACTIVE_READ_BIAS & RWSEM_ACTIVE_MASK == 1
- RWSEM_ACTIVE_WRITE_BIAS & RWSEM_ACTIVE_MASK == 1
- RWSEM_WAITING_BIAS & RWSEM_ACTIVE_MASK == 0
- RWSEM_ACTIVE_WRITE_BIAS < 0 and RWSEM_WAITING_BIAS < 0
In addition, the rwsem count will be < RWSEM_WAITING_BIAS only if there are any active writers (though we don't make use of this property so far).
--- arch/x86/include/asm/rwsem.h | 32 +++++++++++++++++++------------- arch/x86/lib/rwsem_64.S | 4 ++-- arch/x86/lib/semaphore_32.S | 4 ++-- 3 files changed, 23 insertions(+), 17 deletions(-)
arch/x86/include/asm/rwsem.h +19 -13
@@ -16,11 +16,10 @@* if there are writers (and maybe) readers waiting (in which case it goes to * sleep). * -* The value of WAITING_BIAS supports up to 32766 waiting processes. This can -* be extended to 65534 by manually checking the whole MSW rather than relying -* on the S flag. +* The WRITE_BIAS value supports up to 32767 processes simultaneously +* trying to acquire a write lock. * -* The value of ACTIVE_BIAS supports up to 65535 active processes. +* The value of ACTIVE_MASK supports up to 32767 active processes. * * This should be totally fair - if anything is waiting, a process that wants a * lock will go to the back of the queue. When the currently active lock is
@@ -62,17 +61,23 @@ extern asmregparm struct rw_semaphore ** for 64 bits. */ +#ifdef CONFIG_X86_64 -# define RWSEM_ACTIVE_MASK 0xffffffffL +# define RWSEM_UNLOCKED_VALUE 0x0000000000000000L +# define RWSEM_ACTIVE_MASK 0x000000007fffffffL +# define RWSEM_ACTIVE_READ_BIAS 0x0000000000000001L +# define RWSEM_ACTIVE_WRITE_BIAS 0xffffffff00000001L +# define RWSEM_WAITING_BIAS 0xffffffff80000000L +# define RWSEM_WAITING_MASK 0x0000000080000000L#else -# define RWSEM_ACTIVE_MASK 0x0000ffffL +# define RWSEM_UNLOCKED_VALUE 0x00000000L +# define RWSEM_ACTIVE_MASK 0x00007fffL +# define RWSEM_ACTIVE_READ_BIAS 0x00000001L +# define RWSEM_ACTIVE_WRITE_BIAS 0xffff0001L +# define RWSEM_WAITING_BIAS 0xffff8000L +# define RWSEM_WAITING_MASK 0x00008000L#endif -#define RWSEM_UNLOCKED_VALUE 0x00000000L -#define RWSEM_ACTIVE_BIAS 0x00000001L -#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1) -#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS -#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)typedef signed long rwsem_count_t ;
@@ -240,7 +245,8 @@ static inline void __downgrade_write(struct rw_semaphore *sem)"1:\n\t" "# ending __downgrade_write\n" : "+m" (sem->count) - : "a" (sem), "er" (-RWSEM_WAITING_BIAS) + : "a" (sem), +" er" (RWSEM_ACTIVE_READ_BIAS - RWSEM_ACTIVE_WRITE_BIAS) : "memory", "cc"); }
@@ -277,7 +283,7 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)static inline int rwsem_is_contended (struct rw_semaphore *sem) { -return (sem->count <0 ); +return (sem->count &RWSEM_WAITING_MASK ) !=0 ; }#endif /* __KERNEL__ */
arch/x86/lib/rwsem_64.S +2 -2
@@ -60,8 +60,8 @@ ENTRY(call_rwsem_down_write_failed) ENDPROC(call_rwsem_down_write_failed) ENTRY(call_rwsem_wake) - decl %edx /* do nothing if still outstanding active readers */ - jnz 1f + cmpl $0x80000001, %edx + jne 1f /* do nothing unless there are waiters and no active threads */ save_common_regs movq %rax,%rdi call rwsem_wake
arch/x86/lib/semaphore_32.S +2 -2
@@ -103,8 +103,8 @@ ENTRY(call_rwsem_down_write_failed) ENTRY(call_rwsem_wake) CFI_STARTPROC - decw %dx /* do nothing if still outstanding active readers */ - jnz 1f + cmpw $0x8001, %dx + jne 1f /* do nothing unless there are waiters and no active threads */ push %ecx CFI_ADJUST_CFA_OFFSET 4 CFI_REL_OFFSET ecx,0
Signature · 8 lines
1.7.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 6/6] x86 rwsem: more precise rwsem_is_contended() implementation
Quoted 3 lines · Michel Lespinasse
We would like
rwsem_is_contended()to return true only once a contending writer has had a chance to insert itself onto the rwsem wait queue. To that end, we need to differenciate between active and queued writers.
So you're only considering writer-writer contention? Not writer-reader and reader-writer contention?
I'd argue rwsem_is_contended() should return true if there is _any_ blocked task, be it a read or a writer.
If you want something else call it something else, like rwsem_is_write_contended() (there's a pending writer), or rwsem_is_read_contended() (there's a pending reader).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/6] x86 rwsem: more precise rwsem_is_contended() implementation
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Michel Lespinasse
We would like
rwsem_is_contended()to return true only once a contending writer has had a chance to insert itself onto the rwsem wait queue. To that end, we need to differenciate between active and queued writers.So you're only considering writer-writer contention? Not writer-reader and reader-writer contention?
Quoted 6 lines
I'd argue
rwsem_is_contended()should return true if there is _any_ blocked task, be it a read or a writer.If you want something else call it something else, like
rwsem_is_write_contended()(there's a pending writer), orrwsem_is_read_contended()(there's a pending reader).
I said 'a contending writer' because in my use case, the rwsem was being held for read, and other readers would get admitted in (so they don't count as contention).
The code as written will work in the more general case - it tells if there is another thread queued waiting for the rwsem to be released. It can't tell you however if that thread is a reader or a writer (but if mmap_sem is already held for read, then the contending thread must be a writer).
Signature · 2 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies.
Hi David,
I forgot to add you on the original submission, but I think you'd be best qualified to look at the two patches implementing rwsem_is_contended()...
Quoted 26 lines · Michel Lespinasse
Currently
mlock()holdsmmap_semin exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time, during which various commands such as 'ps auxw' will block. This makes sysadmins unhappy:real 14m36.232s
user 0m0.003s
sys 0m0.015s
(output from 'time ps auxw' while a 20GB file was being mlocked without being previously preloaded into page cache)I propose that
mlock()could releasemmap_semafter theVM_LOCKEDbits have been set in all appropriate VMAs. Then a second pass could be done to actually mlock the pages, in small batches, releasingmmap_semwhen we block on disk access or when we detect some contention.Patches are against v2.6.37-rc4 plus my patches to avoid mlock dirtying (presently queued in -mm).
Michel Lespinasse (6):
mlock: only holdmmap_semin shared mode when faulting in pages
mm: addFOLL_MLOCKfollow_pageflag.
mm: moveVM_LOCKEDcheck to__mlock_vma_pages_range()
rwsem: implementrwsem_is_contended()
mlock: do not holdmmap_semfor extended periods of time x86 rwsem: more preciserwsem_is_contended()implementation
Signature · 2 lines
Michel "Walken" Lespinasse A program is never fully debugged until the last user dies.