in_atomic doesn't count local_irq_disable?
Quoted 2 lines · Srivatsa Vaddagiri
This is basically because
down_readwas called with interrupts disabled ..__might_sleepwas "unable" to dump the stack of callers which lead to this problem ..
What do you mean with unable? Could you post what was printed?
I guess it's a get_user within either spin_lock_irq() or local_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated --
Manfred
Quoted 2 lines · Manfred Spraul
What do you mean with unable? Could you post what was printed?
All I used to get was :
"Debug: sleeping function called from invalid context at include/linux/rwsem.h:45in_atomic: 0, irqs_disabled(): 1
Call Trace:"
That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.
I now inserted some printk's in do_page_fault
to print regs->eip before calling down_read i.e:
/*
* If we're in an interrupt, have no user context or are running in an
* atomic region then we must not take the fault..
*/
if (in_atomic() || !mm)
goto bad_area_nosemaphore;+ if (irqs_disabled()) {
+ printk("BAD Access at (EIP) %08lx\n", regs->eip);
+ printk("Bad Access at virtual address %08lx\n",address);
+ }
down_read(&mm->mmap_sem);
This is what I got now when I reran my stress test:
BAD Access at (EIP) c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context at include/linux/rwsem.h:47 in_atomic():0, irqs_disabled():1
Call Trace: [<c011fd66>] __might_sleep+0x86/0x90 [<c01378f6>] module_unload_free+0x36/0xe0 [<c011b889>] do_page_fault+0xc9/0x573 [<c013f1df>] buffered_rmqueue+0x10f/0x120 [<c013f2ba>] __alloc_pages+0xca/0x360 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0 [<c011b7c0>] do_page_fault+0x0/0x573 [<c01378f6>] module_unload_free+0x36/0xe0 [<c0109d6d>] error_code+0x2d/0x38 [<01010101>]
BAD Access at (EIP) c0139934
Bad Access at virtual address 0101011f
The first EIP (c011c1b5) is inside search_extable!! The second EIP (c0139934) is inside get_ksymbol() ...
I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..
The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened in search_extable when looking up some module exception tables(?) ..Because search_module_extables() calls search_extable() with interrupts disabled ..
I think this points to some module unload (race) issues during hotplug ..
Rusty, any comments?
Quoted 2 lines
I guess it's a
get_userwithin eitherspin_lock_irq()orlocal_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated
Given the context above, I feel it would be correct fordo_page_fault() to avoid calling down_read() when IRQs are
disabled and instead just branch to bad_nosemaphore.
(as Rusty seems to concur) .. However, schedule() doesn't
seem to actually trap the case when it is called with interrupts disabled (as using local_irq_disable)?
Signature · 5 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560033
BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
Hi,
in_atomic() doesn't seem to return true
in code sections where IRQ's have been disabled (using local_irq_disable).
As a result, I think do_page_fault() on x86 needs to be updated to note this fact:
fault.c Wed Dec 31 18:35:02 2003 renamed from fault.c.org Wed Dec 31 18:34:18 2003 +1 -1
@@ -259,7 +259,7 @@ * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault.. */ - if (in_atomic() || !mm) + if (in_atomic() || irqs_disabled() || !mm) goto bad_area_nosemaphore; down_read(&mm->mmap_sem);
For ex: cosider some kernel code like this:
local_irq_disable(); /* Do something which leads to a page fault */ local_irq_enable();
Such a code is buggy and should result in a oops(?) when the page fault happens.
Without the above patch, the page fault handler will try acquiring the mmap_sem semaphore and can potentially sleep inside schedule (with IRQs disabled).
Also isn't the same change required in scheduler()
where it whines if called from a code section which is supposed to be atomic.
sched.c Wed Dec 31 18:49:43 2003 renamed from sched.c.org Wed Dec 31 18:49:10 2003 +1 -1
@@ -1507,7 +1507,7 @@ * Otherwise, whine if we are scheduling when we should not be. */ if (likely(!(current->state & (TASK_DEAD | TASK_ZOMBIE)))) { - if (unlikely(in_atomic())) { + if (unlikely(in_atomic() || irqs_disabled())) { printk(KERN_ERR "bad: scheduling while atomic!\n"); dump_stack(); }
Quoted 106 lines · Srivatsa Vaddagiri
Quoted 2 lines · Manfred Spraul
What do you mean with unable? Could you post what was printed?
All I used to get was :
"Debug: sleeping function called from invalid context at
include/linux/rwsem.h:45in_atomic: 0,irqs_disabled(): 1Call Trace:"That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.
I now inserted some printk's in
do_page_fault
to printregs->eipbefore callingdown_readi.e:/* * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault.. */ if (in_atomic() || !mm) goto bad_area_nosemaphore;(quoted diff) +4
-0+ if (irqs_disabled()) { + printk("BAD Access at (EIP) %08lx\n", regs->eip); + printk("Bad Access at virtual address %08lx\n",address); + } down_read(&mm->mmap_sem);This is what I got now when I reran my stress test:
BAD Access at (EIP)
c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context atinclude/linux/rwsem.h:47in_atomic():0, irqs_disabled():1Call Trace: [<c011fd66>] __might_sleep+0x86/0x90 [<c01378f6>] module_unload_free+0x36/0xe0 [<c011b889>] do_page_fault+0xc9/0x573 [<c013f1df>] buffered_rmqueue+0x10f/0x120 [<c013f2ba>] __alloc_pages+0xca/0x360 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0 [<c011b7c0>] do_page_fault+0x0/0x573 [<c01378f6>] module_unload_free+0x36/0xe0 [<c0109d6d>] error_code+0x2d/0x38 [<01010101>]BAD Access at (EIP)
c0139934
Bad Access at virtual address0101011fThe first EIP (
c011c1b5) is insidesearch_extable!! The second EIP (c0139934) is insideget_ksymbol()...I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..
The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened insearch_extablewhen looking up some module exception tables(?) ..Becausesearch_module_extables()callssearch_extable()with interrupts disabled ..I think this points to some module unload (race) issues during hotplug ..
Rusty, any comments?
Quoted 2 lines
I guess it's a
get_userwithin eitherspin_lock_irq()orlocal_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updatedGiven the context above, I feel it would be correct for
do_page_fault()to avoid callingdown_read()when IRQs are
disabled and instead just branch tobad_nosemaphore.
(as Rusty seems to concur) .. However,schedule()doesn't
seem to actually trap the case when it is called with interrupts disabled (as usinglocal_irq_disable)?Signature · 16 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560033 ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ lhcs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/lhcs-devel
Signature · 5 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560017
Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
Quoted 7 lines · Srivatsa Vaddagiri
in_atomic()doesn't seem to return true
in code sections where IRQ's have been disabled (usinglocal_irq_disable).As a result, I think
do_page_fault()on x86 needs to be updated to note this fact:
NO.
Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.
Also, there are broken old drivers that potentially have interrupts disabled, and we shouldn't just oops them. We should have a warning, but we already do have that: that's what "might_sleep()" does.
So something like this may be appropriate at some point, but not in this format. At the very least you absolutely _have_ to check for user mode (possibly in the same place where we now have that
/* It's safe to allow irq's after cr2 has been saved */
comment).
Lnus
Quoted 11 lines
fault.c Wed Dec 31 18:35:02 2003 renamed from fault.c.org Wed Dec 31 18:34:18 2003 +1
-1@@ -259,7 +259,7 @@ * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault.. */ - if (in_atomic() || !mm) + if (in_atomic() || irqs_disabled() || !mm) goto bad_area_nosemaphore; down_read(&mm->mmap_sem);
Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
Hi!
Quoted 4 lines
Quoted 4 lines
in_atomic()doesn't seem to return true
in code sections where IRQ's have been disabled (usinglocal_irq_disable).
Quoted 8 lines
Quoted 2 lines
As a result, I think
do_page_fault()on x86 needs to be updated to note this fact:NO.
Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.
If user program causes page fault with interrupts disabled, it is certainly buggy, right?
Either the user program does not really need irq disabled or it does need that but page fault just broke its guarantees (=> severe problems ahead).
In both cases there's user program that needs fixing.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
Quoted 7 lines · Pavel Machek
Quoted 4 lines
Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.
If user program causes page fault with interrupts disabled, it is certainly buggy, right?
No.
It may do a best-effort thing. It may also do a best-_performance_ thing, in leaving interrupts disabled over a piece of code that doesn't care, knowing that disabling interrupts is expensive. Or it may just be a simple case of simplicity: disable interrupts over the whole region, knowing that only a part of it matters.
It by no means is automatically a bug. And it unquestionably _does_ happen. We used to warn about it. We stopped.
Linus
Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
In reply to Pavel MachekQuoted 7 lines · Pavel Machek
Quoted 4 lines
Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.
If user program causes page fault with interrupts disabled, it is certainly buggy, right?
No.
It may do a best-effort thing. It may also do a best-_performance_ thing, in leaving interrupts disabled over a piece of code that doesn't care, knowing that disabling interrupts is expensive. Or it may just be a simple case of simplicity: disable interrupts over the whole region, knowing that only a part of it matters.
It by no means is automatically a bug. And it unquestionably _does_ happen. We used to warn about it. We stopped.
Linus
Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]
In reply to Linus TorvaldsHi!
Quoted 4 lines
Quoted 4 lines
in_atomic()doesn't seem to return true
in code sections where IRQ's have been disabled (usinglocal_irq_disable).
Quoted 8 lines
Quoted 2 lines
As a result, I think
do_page_fault()on x86 needs to be updated to note this fact:NO.
Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.
If user program causes page fault with interrupts disabled, it is certainly buggy, right?
Either the user program does not really need irq disabled or it does need that but page fault just broke its guarantees (=> severe problems ahead).
In both cases there's user program that needs fixing.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside list_for_each_entry macros.
For now I have disabled the x86 prefetch function to do nothing.
The test seems to run fine so far w/o any of the page faults I was experiencing. Will update
at the end of the overnight run if I hit the problem again.
Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?
Quoted 106 lines · Srivatsa Vaddagiri
Quoted 2 lines · Manfred Spraul
What do you mean with unable? Could you post what was printed?
All I used to get was :
"Debug: sleeping function called from invalid context at
include/linux/rwsem.h:45in_atomic: 0,irqs_disabled(): 1Call Trace:"That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.
I now inserted some printk's in
do_page_fault
to printregs->eipbefore callingdown_readi.e:/* * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault.. */ if (in_atomic() || !mm) goto bad_area_nosemaphore;(quoted diff) +4
-0+ if (irqs_disabled()) { + printk("BAD Access at (EIP) %08lx\n", regs->eip); + printk("Bad Access at virtual address %08lx\n",address); + } down_read(&mm->mmap_sem);This is what I got now when I reran my stress test:
BAD Access at (EIP)
c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context atinclude/linux/rwsem.h:47in_atomic():0, irqs_disabled():1Call Trace: [<c011fd66>] __might_sleep+0x86/0x90 [<c01378f6>] module_unload_free+0x36/0xe0 [<c011b889>] do_page_fault+0xc9/0x573 [<c013f1df>] buffered_rmqueue+0x10f/0x120 [<c013f2ba>] __alloc_pages+0xca/0x360 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0 [<c011b7c0>] do_page_fault+0x0/0x573 [<c01378f6>] module_unload_free+0x36/0xe0 [<c0109d6d>] error_code+0x2d/0x38 [<01010101>]BAD Access at (EIP)
c0139934
Bad Access at virtual address0101011fThe first EIP (
c011c1b5) is insidesearch_extable!! The second EIP (c0139934) is insideget_ksymbol()...I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..
The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened insearch_extablewhen looking up some module exception tables(?) ..Becausesearch_module_extables()callssearch_extable()with interrupts disabled ..I think this points to some module unload (race) issues during hotplug ..
Rusty, any comments?
Quoted 2 lines
I guess it's a
get_userwithin eitherspin_lock_irq()orlocal_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updatedGiven the context above, I feel it would be correct for
do_page_fault()to avoid callingdown_read()when IRQs are
disabled and instead just branch tobad_nosemaphore.
(as Rusty seems to concur) .. However,schedule()doesn't
seem to actually trap the case when it is called with interrupts disabled (as usinglocal_irq_disable)?Signature · 16 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560033 ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ lhcs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/lhcs-devel
Signature · 5 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560017
Quoted 4 lines · Srivatsa Vaddagiri
More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside
list_for_each_entrymacros.
Quoted 9 lines
For now I have disabled the x86 prefetch function
to do nothing.The test seems to run fine so far w/o any of the page faults I was experiencing. Will update at the end of the overnight run if I hit the problem again.
Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?
Hmm. Perhaps prefetch updates CR2?
We know already that the CR2 is not directly linked to the page fault interrupt - if a page fault happens at the same time as a higher priority event (iirc hw interrupt), then CR2 is updated and the higher priority event is handled. That prevents Linux from using CR2 to store the cpu number - only netware can do that, because netware never causes paging faults.
Could you write a test module that reads cr2, executes a few prefetch instructions and then checks if cr2 changed? I won't have access to my P3 SMP system in the next few days.
--
Manfred
Quoted 3 lines · Manfred Spraul
Could you write a test module that reads cr2, executes a few prefetch instructions and then checks if cr2 changed? I won't have access to my P3 SMP system in the next few days.
Hi Manfred,
I wrote a test module and found that CR2 remains same across the
prefetch. The module source I used is as below. Note that I had to used "my_prefetch" because the original prefetch (in asm/processor.h) has been disabled in my tree to do nothing.
inline void my_prefetch(const void *x)
{
alternative_input(ASM_NOP4,
"prefetchnta (%1)",
X86_FEATURE_XMM,
"r" (x));
}int array[10];
static int __init dummy_init_module(void)
{
unsigned long address;
int i=0;
int x;
/* get the address */
__asm__("movl %%cr2,%0":"=r" (address));
printk ("CR2 before prefetch is %x \n", address);
for (i=0; i<10; ++i)
my_prefetch(array+i);
for (i=0; i<10; ++i)
x = *(array+i);
/* get the address */
__asm__("movl %%cr2,%0":"=r" (address));
printk ("CR2 after prefetch is %x \n", address);
return 0;}
static void __exit dummy_cleanup_module(void)
{
}
module_init(dummy_init_module);module_exit(dummy_cleanup_module);MODULE_LICENSE("GPL");
Output of the above printk is :
CR2 before prefetch is 40017000
CR2 after prefetch is 40017000
Signature · 5 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560017
Quoted 4 lines · Srivatsa Vaddagiri
More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside
list_for_each_entrymacros.
Quoted 9 lines
For now I have disabled the x86 prefetch function to do nothing.
The test seems to run fine so far w/o any of the page faults I was experiencing. Will update
at the end of the overnight run if I hit the problem again.Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?
Even after disabling prefetch, I continue
to hit page-faults.
With prefetch disabled, it _always_
traps because of trying to dereference a NULL pointer in a list-head. If I break-in into the debugger, the list-head is actually valid (no NULL pointer is present) and hence I don't understand why it read a NULL pointer value in a list head.
With prefetch enabled it traps
because of fetch from arbitrary (random) addresses.
So I am no longer sure if this is a prefetch issue or/and a hotplug issue. I'm continuing to investigate and will post any new observation I make here.
FYI, the /proc/cpuinfo output on my SMP system is as below:
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.730
cache size : 1024 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1376.25
processor : 1vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73
processor : 2vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73
processor : 3vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73
Signature · 5 lines
Thanks and Regards, Srivatsa Vaddagiri, Linux Technology Center, IBM Software Labs, Bangalore, INDIA - 560017