core kernel fixes
Linus,
Please pull the latest core-fixes-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git core-fixes-for-linus
Thanks,
Ingo
------------------>
Joerg Roedel (2):
dma-debug: check for sg_call_ents in best-fit algorithm too
dma-debug: be more careful when building reference entries
Peter Zijlstra (1):
lockdep: Select frame pointers on x86
Thomas Gleixner (1):
futex: Fix the write access fault problem for real
kernel/futex.c | 51 +++++++++++-------- lib/Kconfig.debug | 2 +- lib/dma-debug.c | 149 +++++++++++++++++++++++++++++++++++------------------ 3 files changed, 129 insertions(+), 73 deletions(-)
kernel/futex.c +30 -21
@@ -284,6 +284,31 @@ void put_futex_key(int fshared, union futex_key *key)drop_futex_key_refs (key); } +/* +* get_user_writeable - get user page and verify RW access +* @uaddr: pointer to faulting user space address +* +* We cannot write to the user space address and get_user just faults +* the page in, but does not tell us whether the mapping is writeable. +* +* We can not rely on access_ok() for private futexes as it is just a +* range check and we can neither rely on get_user_pages() as there +* might be a mprotect(PROT_READ) for that mapping after +* get_user_pages() and before the fault in the atomic write access. +*/ +static int get_user_writeable (u32 __user *uaddr) +{ +unsigned long addr = (unsigned long )uaddr; +struct page *page; +int ret; + + ret =get_user_pages_fast (addr,1 ,1 , &page); +if (ret >0 ) +put_page (page); + +return ret; +} +/** * futex_top_waiter() - Return the highest priority waiter on a futex * @hb: the hash bucket the futex_q's reside in
@@ -896,7 +921,6 @@ retry: retry_private: op_ret =futex_atomic_op_inuser (op, uaddr2);if (unlikely (op_ret <0 )) { -u32 dummy;double_unlock_hb (hb1, hb2);
@@ -914,7 +938,7 @@ retry_private: goto out_put_keys; } - ret =get_user (dummy, uaddr2); + ret =get_user_writeable (uaddr2);if (ret) goto out_put_keys;
@@ -1204,7 +1228,7 @@ retry_private:double_unlock_hb (hb1, hb2);put_futex_key (fshared, &key2);put_futex_key (fshared, &key1); - ret =get_user (curval2, uaddr2); + ret =get_user_writeable (uaddr2);if (!ret) goto retry; goto out;
@@ -1482,7 +1506,7 @@ retry: handle_fault:spin_unlock (q->lock_ptr); - ret =get_user (uval, uaddr); + ret =get_user_writeable (uaddr);spin_lock (q->lock_ptr);
@@ -1807,7 +1831,6 @@ static int futex_lock_pi(u32 __user *uaddr, int fshared, {struct hrtimer_sleeper timeout, *to =NULL ;struct futex_hash_bucket *hb; -u32 uval;struct futex_q q;int res, ret;
@@ -1909,16 +1932,9 @@ out:return ret != -EINTR ? ret : -ERESTARTNOINTR ; uaddr_faulted: -/* -* We have to r/w *(int __user *)uaddr, and we have to modify it -* atomically. Therefore, if we continue to fault after get_user() -* below, we need to handle the fault ourselves, while still holding -* the mmap_sem. This can occur if the uaddr is under contention as -* we have to drop the mmap_sem in order to call get_user(). -*/ queue_unlock (&q, hb); - ret =get_user (uval, uaddr); + ret =get_user_writeable (uaddr);if (ret) goto out_put_key;
@@ -2013,17 +2029,10 @@ out:return ret; pi_faulted: -/* -* We have to r/w *(int __user *)uaddr, and we have to modify it -* atomically. Therefore, if we continue to fault after get_user() -* below, we need to handle the fault ourselves, while still holding -* the mmap_sem. This can occur if the uaddr is under contention as -* we have to drop the mmap_sem in order to call get_user(). -*/ spin_unlock (&hb->lock);put_futex_key (fshared, &key); - ret =get_user (uval, uaddr); + ret =get_user_writeable (uaddr);if (!ret) goto retry;
lib/Kconfig.debug +1 -1
@@ -440,7 +440,7 @@ config LOCKDEP bool depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT select STACKTRACE - select FRAME_POINTER if !X86 && !MIPS && !PPC && !ARM_UNWIND && !S390 + select FRAME_POINTER if !MIPS && !PPC && !ARM_UNWIND && !S390 select KALLSYMS select KALLSYMS_ALL
lib/dma-debug.c +98 -51
284 lines not on this page · raw message
Quoted 2 lines · Ingo Molnar
Please pull the latest core-fixes-for-linus git tree from:
I need to think about this one.
Quoted 14 lines
kernel/futex.c +14
-0+/* +* get_user_writeable - get user page and verify RW access +* @uaddr: pointer to faulting user space address +* +* We cannot write to the user space address and get_user just faults +* the page in, but does not tell us whether the mapping is writeable. +* +* We can not rely on access_ok() for private futexes as it is just a +* range check and we can neither rely on get_user_pages() as there +* might be a mprotect(PROT_READ) for that mapping after +* get_user_pages() and before the fault in the atomic write access. +*/ +static int get_user_writeable (u32 __user *uaddr) +{
kernel/futex.c void put_futex_key(int fshared, union futex_key *key) · line 310
Quoted 10 lines
kernel/futex.c +10
-0+unsigned long addr = (unsigned long )uaddr; +struct page *page; +int ret; + + ret =get_user_pages_fast (addr,1 ,1 , &page); +if (ret >0 ) +put_page (page); + +return ret; +}
This is some seriously crazy sh*t, man. What drugs were you on, and are the police on to you? Because whatever drugs they were, I seriously doube they are legal even with a prescription.
There's somethign wrong in futex land. This whole retry crap has been so incredibly broken so many times, and this particular fix looks so horribly ugly that I really need to ask people: "is the loop really sane?"
I also think that the above is a singularly stupid way to do what you want done. It's slow, it's nasty, it's complicated.
On x86, the natural way to do what you want done is ONE SINGLE INSTRUCTION! As far as I can tell, the above crazy function is 100% equivalent to this:
asm __inline__("lock ; addl $0,%0":"+m" (*uaddr): :"memory", "cc");which really makes me think that using "get_user_pages_fast()" for it is some truly crazy crap.
Sure, we don't have any architecture-generic way to do that, but still..
Linus
Quoted 9 lines · Linus Torvalds
On x86, the natural way to do what you want done is ONE SINGLE INSTRUCTION! As far as I can tell, the above crazy function is 100% equivalent to this:
asm __inline__("lock ; addl $0,%0":"+m" (*uaddr): :"memory", "cc");which really makes me think that using "
get_user_pages_fast()" for it is some truly crazy crap.
We could also take the opposite approach - knowing that this is called only when the page doesn't exist, and just doing
down_read(mmap_sem) vma = find_vma(..) ret = VM_FAULT_ERROR; if (vma && vma->vm_start <= address) ret = handle_mm_fault(mm, vma, address, 1); up_read(mmap_sem); return (ret & VM_FAULT_ERROR) ? -EFAULT : 0;
or something like that. Again, that looks saner than using get_user_pages() for this and then dropping the page.
Linus
Quoted 22 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
On x86, the natural way to do what you want done is ONE SINGLE INSTRUCTION! As far as I can tell, the above crazy function is 100% equivalent to this:
asm __inline__("lock ; addl $0,%0":"+m" (*uaddr): :"memory", "cc");which really makes me think that using "
get_user_pages_fast()" for it is some truly crazy crap.We could also take the opposite approach - knowing that this is called only when the page doesn't exist, and just doing
down_read(mmap_sem) vma = find_vma(..) ret = VM_FAULT_ERROR; if (vma && vma->vm_start <= address) ret = handle_mm_fault(mm, vma, address, 1); up_read(mmap_sem); return (ret & VM_FAULT_ERROR) ? -EFAULT : 0;
Quoted 2 lines
or something like that. Again, that looks saner than using
get_user_pages()for this and then dropping the page.
We'll sort this out tomorrow, sorry about this.
( The other embarrasing bit is that i flagged this commit as bad
nine days ago - during review i noticed the same bad pattern of a pointless get-put cycle ... but the commit stuck around and forgot about it. Sloppy. )
Ingo
Quoted 22 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
On x86, the natural way to do what you want done is ONE SINGLE INSTRUCTION! As far as I can tell, the above crazy function is 100% equivalent to this:
asm __inline__("lock ; addl $0,%0":"+m" (*uaddr): :"memory", "cc");which really makes me think that using "
get_user_pages_fast()" for it is some truly crazy crap.We could also take the opposite approach - knowing that this is called only when the page doesn't exist, and just doing
down_read(mmap_sem) vma = find_vma(..) ret = VM_FAULT_ERROR; if (vma && vma->vm_start <= address) ret = handle_mm_fault(mm, vma, address, 1); up_read(mmap_sem); return (ret & VM_FAULT_ERROR) ? -EFAULT : 0;
Quoted 2 lines
or something like that. Again, that looks saner than using
get_user_pages()for this and then dropping the page.
Hmm. The main reason why we switched to get_user_pages_fast() in the futex code is to avoid mmap_sem contention which was observed as a real big performance problem especially with those horrible JavaVM applications.
As a fallout of this we got rid of the private find_vma / handle_mm_fault magic (as above) in the futex code which mm folks frowned upon for quite a while. Unfortunately we got it wrong :(
I agree that in the fault path we might go back to the mmap_sem version, but I want to avoid it when possible. At least we need to run some of those horrible JavaVM apps to see whether it matters or not.
OTOH, I really wonder whether we can simplify the whole logic when we keep the page reference for the atomic access to the user space address, but I still have doubts about the life time rules of all this. Also I need to find out how this would affect the fast path optimization of private futexes where we avoid the lookup in the first place.
Thanks,
tglx
Quoted 5 lines · Thomas Gleixner
Hmm. The main reason why we switched to
get_user_pages_fast()in the futex code is to avoidmmap_semcontention which was observed as a real big performance problem especially with those horrible JavaVM applications.
Not relevant.
get_user_pages_fast() takes the mmap_sem for the case where it needs to fault things in too.
So assuming the _only_ reason this thing is called is because we failed earlier when doing the futex_atomic_op_inuser(), then you're basically guaranteed that the "fast" case of get_user_pages_fast() is never actually taken, since we already know that the page tables aren't amenable to an atomic access.
And as far as I can tell, that is indeed the only case where you use that 'get_user_writeable()' thing. You've had futex_atomic_op_inuser() fail, and need to repeat. No?
Quoted 3 lines
As a fallout of this we got rid of the private
find_vma/handle_mm_faultmagic (as above) in the futex code which mm folks frowned upon for quite a while. Unfortunately we got it wrong :(
Sure. But "get_user_pages_fast()" really is the wrong thing. You're not at all interested in the user pages. You're interested in making sure that the page is atomically writable, and nothing else. Right?
Which is why I said that "lock ; addl $0,(mem)" would be a _single_ instruction, and do everything that your "get_user_pages_fast()" hack would do. If the fault is unlikely, that would be a better operation. I just don't think the fault is unlikely, I suspect it happens every time.
Linus
Quoted 4 lines · Linus Torvalds
And as far as I can tell, that is indeed the only case where you use that '
get_user_writeable()' thing. You've hadfutex_atomic_op_inuser()fail, and need to repeat. No?
I just checked. Yes, this code is _only_ entered when an atomic op returned EFAULT. IOW, we absolutely know that the page tables are not set up for writability, and thus that the "fast" case will never ever trigger.
(Ok, in theory you could have some other thread writing to that page at the same time and handle the page fault and making it writable, but in practice that's not really relevant).
So just doing a "make_sure_its_writable()" and using handle_fault() is the right thing to do. Because it's what get_user_fast() would have done too, except it would have gone through first the fast case, and failed, then the slow case, and failed the lookup there, and then the slow case would have done that handle_mm_fault() in the end anyway.
In fact, since you're not actually interested in the page, you _could_ just do
get_user_pages(tsk, mm, uaddr, 4, 1, 0, NULL, NULL);
where a NULL "pages" pointer already tells get_user_pages() that you're not interested.
That's at least cleaner than doing a "gup_fast()" (which isn't fast), and then freeing the page that you weren't even interested in.
Linus
Linus,
Quoted 9 lines · Linus Torvalds
So just doing a "
make_sure_its_writable()" and usinghandle_fault()is the right thing to do. Because it's whatget_user_fast()would have done too, except it would have gone through first the fast case, and failed, then the slow case, and failed the lookup there, and then the slow case would have done thathandle_mm_fault()in the end anyway.In fact, since you're not actually interested in the page, you _could_ just do
Quoted 7 lines
get_user_pages(tsk, mm, uaddr, 4, 1, 0, NULL, NULL);where a NULL "pages" pointer already tells
get_user_pages()that you're not interested.That's at least cleaner than doing a "
gup_fast()" (which isn't fast), and then freeing the page that you weren't even interested in.
Yes, you are right. The retry fixup path is after the fault and we should go through handle_mm_fault as long as we do not have a general available nondestructive counterpart of get_user().
I confused myself by twisting my brain whether we can simplify or even get rid of the whole retry business.
Sorry, I did not express myself very well - looking for more than an hour into the futex code definitely hurts your brain. It's worse than the drugs you suspected we're on. :)
Thanks,
tglx