Skip to contentSkip to conversation

in_atomic doesn't count local_irq_disable?

DISCUSSION Manfred Spraul 4 participants 12 messages Dec 29, 2003 – Mar 29, 2004

lore
Share & tools
lore
#0 lore raw
Quoted 2 lines · Srivatsa Vaddagiri

This is basically because down_read was called with interrupts disabled .. __might_sleep was "unable" to dump the stack of callers which lead to this problem ..

What do you mean with unable? Could you post what was printed?

I guess it's a get_user within either spin_lock_irq() or local_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated --
Manfred

#1 lore raw
Quoted 2 lines · Manfred Spraul

What do you mean with unable? Could you post what was printed?

All I used to get was :

"Debug: sleeping function called from invalid context at include/linux/rwsem.h:45
in_atomic: 0, irqs_disabled(): 1

Call Trace:"

That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.

I now inserted some printk's in do_page_fault
to print regs->eip before calling down_read i.e:

        /*
         * If we're in an interrupt, have no user context or are running in an
         * atomic region then we must not take the fault..
         */
        if (in_atomic() || !mm)
                goto bad_area_nosemaphore;

+ if (irqs_disabled()) {
+ printk("BAD Access at (EIP) %08lx\n", regs->eip);
+ printk("Bad Access at virtual address %08lx\n",address);
+ }

        down_read(&mm->mmap_sem);

This is what I got now when I reran my stress test:

BAD Access at (EIP) c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context at include/linux/rwsem.h:47 in_atomic():0, irqs_disabled():1

Call Trace:
 [<c011fd66>] __might_sleep+0x86/0x90
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c011b889>] do_page_fault+0xc9/0x573
 [<c013f1df>] buffered_rmqueue+0x10f/0x120
 [<c013f2ba>] __alloc_pages+0xca/0x360
 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0
 [<c011b7c0>] do_page_fault+0x0/0x573
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c0109d6d>] error_code+0x2d/0x38
 [<01010101>] 

BAD Access at (EIP) c0139934
Bad Access at virtual address 0101011f

The first EIP (c011c1b5) is inside search_extable!! The second EIP (c0139934) is inside get_ksymbol() ...

I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..

The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened in search_extable when looking up some module exception tables(?) ..Because search_module_extables() calls search_extable() with interrupts disabled ..

I think this points to some module unload (race) issues during hotplug ..

Rusty, any comments?

Quoted 2 lines

I guess it's a get_user within either spin_lock_irq() or local_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated

Given the context above, I feel it would be correct for
do_page_fault() to avoid calling down_read() when IRQs are
disabled and instead just branch to bad_nosemaphore.
(as Rusty seems to concur) .. However, schedule() doesn't
seem to actually trap the case when it is called with interrupts disabled (as using local_irq_disable)?

Signature · 5 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560033
#2 lore raw

BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

Hi,
in_atomic() doesn't seem to return true
in code sections where IRQ's have been disabled (using local_irq_disable).

As a result, I think do_page_fault() on x86 needs to be updated to note this fact:

fault.c Wed Dec 31 18:35:02 2003 renamed from fault.c.org Wed Dec 31 18:34:18 2003 +1 -1
@@ -259,7 +259,7 @@
         * If we're in an interrupt, have no user context or are running in an
         * atomic region then we must not take the fault..
         */
-       if (in_atomic() || !mm)
+       if (in_atomic() || irqs_disabled() || !mm)
                goto bad_area_nosemaphore;
 
        down_read(&mm->mmap_sem);

For ex: cosider some kernel code like this:

	local_irq_disable();

	/* Do something which leads to a page fault */

	local_irq_enable(); 

Such a code is buggy and should result in a oops(?) when the page fault happens.

Without the above patch, the page fault handler will try acquiring the mmap_sem semaphore and can potentially sleep inside schedule (with IRQs disabled).

Also isn't the same change required in scheduler()
where it whines if called from a code section which is supposed to be atomic.

sched.c Wed Dec 31 18:49:43 2003 renamed from sched.c.org Wed Dec 31 18:49:10 2003 +1 -1
@@ -1507,7 +1507,7 @@
         * Otherwise, whine if we are scheduling when we should not be.
         */
        if (likely(!(current->state & (TASK_DEAD | TASK_ZOMBIE)))) {
-               if (unlikely(in_atomic())) {
+               if (unlikely(in_atomic() || irqs_disabled())) {
                        printk(KERN_ERR "bad: scheduling while atomic!\n");
                        dump_stack();
                }
Quoted 106 lines · Srivatsa Vaddagiri
Quoted 2 lines · Manfred Spraul

What do you mean with unable? Could you post what was printed?

All I used to get was :

"Debug: sleeping function called from invalid context at include/linux/rwsem.h:45
in_atomic: 0, irqs_disabled(): 1

Call Trace:"

That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.

I now inserted some printk's in do_page_fault
to print regs->eip before calling down_read i.e:

        /*
         * If we're in an interrupt, have no user context or are running in an
         * atomic region then we must not take the fault..
         */
        if (in_atomic() || !mm)
                goto bad_area_nosemaphore;
(quoted diff) +4 -0
+       if (irqs_disabled()) {
+               printk("BAD Access at (EIP) %08lx\n", regs->eip);
+               printk("Bad Access at virtual address %08lx\n",address);
+       }
 
        down_read(&mm->mmap_sem);

This is what I got now when I reran my stress test:

BAD Access at (EIP) c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context at include/linux/rwsem.h:47 in_atomic():0, irqs_disabled():1

Call Trace:
 [<c011fd66>] __might_sleep+0x86/0x90
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c011b889>] do_page_fault+0xc9/0x573
 [<c013f1df>] buffered_rmqueue+0x10f/0x120
 [<c013f2ba>] __alloc_pages+0xca/0x360
 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0
 [<c011b7c0>] do_page_fault+0x0/0x573
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c0109d6d>] error_code+0x2d/0x38
 [<01010101>] 

BAD Access at (EIP) c0139934
Bad Access at virtual address 0101011f

The first EIP (c011c1b5) is inside search_extable!! The second EIP (c0139934) is inside get_ksymbol() ...

I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..

The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened in search_extable when looking up some module exception tables(?) ..Because search_module_extables() calls search_extable() with interrupts disabled ..

I think this points to some module unload (race) issues during hotplug ..

Rusty, any comments?

Quoted 2 lines

I guess it's a get_user within either spin_lock_irq() or local_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated

Given the context above, I feel it would be correct for
do_page_fault() to avoid calling down_read() when IRQs are
disabled and instead just branch to bad_nosemaphore.
(as Rusty seems to concur) .. However, schedule() doesn't
seem to actually trap the case when it is called with interrupts disabled (as using local_irq_disable)?

Signature · 16 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560033


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
lhcs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lhcs-devel
Signature · 5 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
#8 lore raw
In reply to Srivatsa Vaddagiri

More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside list_for_each_entry macros.

For now I have disabled the x86 prefetch function to do nothing.

The test seems to run fine so far w/o any of the page faults I was experiencing. Will update
at the end of the overnight run if I hit the problem again.

Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?

Quoted 106 lines · Srivatsa Vaddagiri
Quoted 2 lines · Manfred Spraul

What do you mean with unable? Could you post what was printed?

All I used to get was :

"Debug: sleeping function called from invalid context at include/linux/rwsem.h:45
in_atomic: 0, irqs_disabled(): 1

Call Trace:"

That's it. Nothing more. Looks like it could not read the stack at that point and hence couldn't dump the stack traceback.

I now inserted some printk's in do_page_fault
to print regs->eip before calling down_read i.e:

        /*
         * If we're in an interrupt, have no user context or are running in an
         * atomic region then we must not take the fault..
         */
        if (in_atomic() || !mm)
                goto bad_area_nosemaphore;
(quoted diff) +4 -0
+       if (irqs_disabled()) {
+               printk("BAD Access at (EIP) %08lx\n", regs->eip);
+               printk("Bad Access at virtual address %08lx\n",address);
+       }
 
        down_read(&mm->mmap_sem);

This is what I got now when I reran my stress test:

BAD Access at (EIP) c011c1b5
Bad Access at virtual address 05050501
Debug: sleeping function called from invalid context at include/linux/rwsem.h:47 in_atomic():0, irqs_disabled():1

Call Trace:
 [<c011fd66>] __might_sleep+0x86/0x90
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c011b889>] do_page_fault+0xc9/0x573
 [<c013f1df>] buffered_rmqueue+0x10f/0x120
 [<c013f2ba>] __alloc_pages+0xca/0x360
 [<c0148d64>] do_anonymous_page+0x1c4/0x1d0
 [<c011b7c0>] do_page_fault+0x0/0x573
 [<c01378f6>] module_unload_free+0x36/0xe0
 [<c0109d6d>] error_code+0x2d/0x38
 [<01010101>] 

BAD Access at (EIP) c0139934
Bad Access at virtual address 0101011f

The first EIP (c011c1b5) is inside search_extable!! The second EIP (c0139934) is inside get_ksymbol() ...

I suspect the second happened when kdb tried decoding the (first) exception address and hence is secondary here ..

The stack trace that follows the first exception seems to be totally bogus(?) .. I suspect the first exception
happened in search_extable when looking up some module exception tables(?) ..Because search_module_extables() calls search_extable() with interrupts disabled ..

I think this points to some module unload (race) issues during hotplug ..

Rusty, any comments?

Quoted 2 lines

I guess it's a get_user within either spin_lock_irq() or local_irq_disable. Without more info about the context, it's difficult to figure out if the page fault handler or the caller should be updated

Given the context above, I feel it would be correct for
do_page_fault() to avoid calling down_read() when IRQs are
disabled and instead just branch to bad_nosemaphore.
(as Rusty seems to concur) .. However, schedule() doesn't
seem to actually trap the case when it is called with interrupts disabled (as using local_irq_disable)?

Signature · 16 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560033


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
lhcs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lhcs-devel
Signature · 5 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
#3 lore raw

Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

In reply to Srivatsa Vaddagiri
Quoted 7 lines · Srivatsa Vaddagiri

in_atomic() doesn't seem to return true
in code sections where IRQ's have been disabled (using local_irq_disable).

As a result, I think do_page_fault() on x86 needs to be updated to note this fact:

NO.

Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.

Also, there are broken old drivers that potentially have interrupts disabled, and we shouldn't just oops them. We should have a warning, but we already do have that: that's what "might_sleep()" does.

So something like this may be appropriate at some point, but not in this format. At the very least you absolutely _have_ to check for user mode (possibly in the same place where we now have that

	/* It's safe to allow irq's after cr2 has been saved */

comment).

Lnus

Quoted 11 lines
fault.c Wed Dec 31 18:35:02 2003 renamed from fault.c.org Wed Dec 31 18:34:18 2003 +1 -1
@@ -259,7 +259,7 @@
         * If we're in an interrupt, have no user context or are running in an
         * atomic region then we must not take the fault..
         */
-       if (in_atomic() || !mm)
+       if (in_atomic() || irqs_disabled() || !mm)
                goto bad_area_nosemaphore;
 
        down_read(&mm->mmap_sem);
#9 lore raw
In reply to Srivatsa Vaddagiri
Quoted 4 lines · Srivatsa Vaddagiri

More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside list_for_each_entry macros.

Quoted 9 lines

For now I have disabled the x86 prefetch function
to do nothing.

The test seems to run fine so far w/o any of the page faults I was experiencing. Will update at the end of the overnight run if I hit the problem again.

Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?

Hmm. Perhaps prefetch updates CR2?
We know already that the CR2 is not directly linked to the page fault interrupt - if a page fault happens at the same time as a higher priority event (iirc hw interrupt), then CR2 is updated and the higher priority event is handled. That prevents Linux from using CR2 to store the cpu number - only netware can do that, because netware never causes paging faults.

Could you write a test module that reads cr2, executes a few prefetch instructions and then checks if cr2 changed? I won't have access to my P3 SMP system in the next few days.

--
Manfred

#10 lore raw
Quoted 3 lines · Manfred Spraul

Could you write a test module that reads cr2, executes a few prefetch instructions and then checks if cr2 changed? I won't have access to my P3 SMP system in the next few days.

Hi Manfred,
I wrote a test module and found that CR2 remains same across the
prefetch. The module source I used is as below. Note that I had to used "my_prefetch" because the original prefetch (in asm/processor.h) has been disabled in my tree to do nothing.

inline void my_prefetch(const void *x)

{
        alternative_input(ASM_NOP4,
                          "prefetchnta (%1)",
                          X86_FEATURE_XMM,
                          "r" (x));
}

int array[10];

static int __init dummy_init_module(void)

{
        unsigned long address;
        int i=0;
        int x;

        /* get the address */
        __asm__("movl %%cr2,%0":"=r" (address));

        printk ("CR2 before prefetch is %x \n", address);

        for (i=0; i<10; ++i)
                my_prefetch(array+i);

        for (i=0; i<10; ++i)
                x = *(array+i);

        /* get the address */
        __asm__("movl %%cr2,%0":"=r" (address));


        printk ("CR2 after prefetch is %x \n", address);


        return 0;

}

static void __exit dummy_cleanup_module(void)
{
}

module_init(dummy_init_module);
module_exit(dummy_cleanup_module);
MODULE_LICENSE("GPL");

Output of the above printk is :

CR2 before prefetch is 40017000
CR2 after prefetch is 40017000

Signature · 5 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
#11 lore raw
In reply to Srivatsa Vaddagiri
Quoted 4 lines · Srivatsa Vaddagiri

More debugging reveals that the page fault happens always while doing a prefetch. The prefetch is present inside list_for_each_entry macros.

Quoted 9 lines

For now I have disabled the x86 prefetch function to do nothing.

The test seems to run fine so far w/o any of the page faults I was experiencing. Will update
at the end of the overnight run if I hit the problem again.

Wonder if prefetch has some issues on Intel x86 (P3) SMP systems?

Even after disabling prefetch, I continue
to hit page-faults.

With prefetch disabled, it _always_
traps because of trying to dereference a NULL pointer in a list-head. If I break-in into the debugger, the list-head is actually valid (no NULL pointer is present) and hence I don't understand why it read a NULL pointer value in a list head.

With prefetch enabled it traps
because of fetch from arbitrary (random) addresses.

So I am no longer sure if this is a prefetch issue or/and a hotplug issue. I'm continuing to investigate and will post any new observation I make here.

FYI, the /proc/cpuinfo output on my SMP system is as below:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.730
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1376.25

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 10
model name : Pentium III (Cascades)
stepping : 1
cpu MHz : 699.726
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1396.73

Signature · 5 lines
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
#4 lore raw

Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

In reply to Linus Torvalds

Hi!

Quoted 4 lines
Quoted 4 lines

in_atomic() doesn't seem to return true
in code sections where IRQ's have been disabled (using local_irq_disable).

Quoted 8 lines
Quoted 2 lines

As a result, I think do_page_fault() on x86 needs to be updated to note this fact:

NO.

Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.

If user program causes page fault with interrupts disabled, it is certainly buggy, right?

Either the user program does not really need irq disabled or it does need that but page fault just broke its guarantees (=> severe problems ahead).

In both cases there's user program that needs fixing.
Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#5 lore raw

Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

Quoted 7 lines · Pavel Machek
Quoted 4 lines

Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.

If user program causes page fault with interrupts disabled, it is certainly buggy, right?

No.

It may do a best-effort thing. It may also do a best-_performance_ thing, in leaving interrupts disabled over a piece of code that doesn't care, knowing that disabling interrupts is expensive. Or it may just be a simple case of simplicity: disable interrupts over the whole region, knowing that only a part of it matters.

It by no means is automatically a bug. And it unquestionably _does_ happen. We used to warn about it. We stopped.

Linus

#7 lore raw

Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

In reply to Linus Torvalds

Hi!

Quoted 4 lines
Quoted 4 lines

in_atomic() doesn't seem to return true
in code sections where IRQ's have been disabled (using local_irq_disable).

Quoted 8 lines
Quoted 2 lines

As a result, I think do_page_fault() on x86 needs to be updated to note this fact:

NO.

Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.

If user program causes page fault with interrupts disabled, it is certainly buggy, right?

Either the user program does not really need irq disabled or it does need that but page fault just broke its guarantees (=> severe problems ahead).

In both cases there's user program that needs fixing.
Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#6 lore raw

Re: BUG in x86 do_page_fault? [was Re: in_atomic doesn't count local_irq_disable?]

In reply to Pavel Machek
Quoted 7 lines · Pavel Machek
Quoted 4 lines

Please don't do this, it will result in some _really_ nasty problems with X and other programs that potentially disable interrupts in user space.

If user program causes page fault with interrupts disabled, it is certainly buggy, right?

No.

It may do a best-effort thing. It may also do a best-_performance_ thing, in leaving interrupts disabled over a piece of code that doesn't care, knowing that disabling interrupts is expensive. Or it may just be a simple case of simplicity: disable interrupts over the whole region, knowing that only a part of it matters.

It by no means is automatically a bug. And it unquestionably _does_ happen. We used to warn about it. We stopped.

Linus