2.6.26.6-rt11: BUGs (sleeping function called from invalid context)
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:31:19 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:19 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:19 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:19 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:19 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:19 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:19 host kernel: =======================
Nov 7 10:31:01 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:01 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:01 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:01 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:01 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:31:01 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:01 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:01 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:01 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:01 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:01 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:01 host kernel: =======================
Nov 7 10:30:58 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:30:58 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:30:58 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:30:58 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:30:58 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:30:58 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:30:58 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:30:58 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:30:58 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:30:58 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:30:58 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:30:58 host kernel: =======================
-- Fernando
Steven Rostedt
reviewerPeter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
As here.
-- Steve
Quoted 63 lines
Nov 7 10:31:19 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:19 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:19 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:19 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:19 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:19 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:19 host kernel: =======================Nov 7 10:31:01 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:01 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:01 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:01 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:01 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:31:01 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:01 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:01 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:01 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:01 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:01 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:01 host kernel: =======================Nov 7 10:30:58 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:30:58 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:30:58 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:30:58 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:30:58 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:30:58 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:30:58 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:30:58 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:30:58 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:30:58 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:30:58 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:30:58 host kernel: =======================-- Fernando
Peter Zijlstra
maintainerQuoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative kmap_atomic implementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?
--- arch/x86/mm/highmem_32.c | 130 ++++++++++++++++++++++++++++++++++++++++++--- include/asm-x86/highmem.h | 11 ---- include/linux/sched.h | 5 ++ kernel/sched.c | 10 ++++ 4 files changed, 137 insertions(+), 19 deletions(-)
arch/x86/mm/highmem_32.c +122 -8
@@ -39,6 +39,12 @@ struct page *kmap_to_page(void *ptr) }EXPORT_SYMBOL_GPL (kmap_to_page);/* PREEMPT_RT converts some modules to use this */ +EXPORT_SYMBOL (kmap); +EXPORT_SYMBOL (kunmap); +EXPORT_SYMBOL (kunmap_virt); + +#ifndef CONFIG_PREEMPT_RT +static void debug_kmap_atomic_prot (enum km_type type) {#ifdef CONFIG_DEBUG_HIGHMEM
@@ -112,11 +118,6 @@ void *__kmap_atomic_prot(struct page *page, enum km_type type, pgprot_t prot)return (void *)vaddr; } -void *__kmap_atomic (struct page *page,enum km_type type) -{ -return kmap_atomic_prot (page, type, kmap_prot); -} -void __kunmap_atomic (void *kvaddr,enum km_type type) {unsigned long vaddr = (unsigned long ) kvaddr &PAGE_MASK ;
@@ -161,6 +162,121 @@ void *__kmap_atomic_pfn(unsigned long pfn, enum km_type type)return (void *) vaddr; } +#else /* CONFIG_PREEMPT_RT */ + +void *__kmap_atomic_prot (struct page *page,enum km_type type,pgprot_t prot) +{ +unsigned long vaddr; + +preempt_disable (); +pagefault_disable (); +if (!PageHighMem (page)) + vaddr = (unsigned long )page_address (page); +else { +enum fixed_addresses idx; +pte_t pte; + + idx = type +KM_TYPE_NR *smp_processor_id (); + vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx); +WARN_ON_ONCE (!pte_none (*(kmap_pte - idx))); + pte =mk_pte (page, prot); + current->kmap_atomic_ptes[idx] = pte; + current->kmap_atomic_count++; +set_pte (kmap_pte - idx, pte); +arch_flush_lazy_mmu_mode (); + } +preempt_enable (); + +return (void *)vaddr; +} + +void __kunmap_atomic (void *kvaddr,enum km_type type) +{ +unsigned long vaddr = (unsigned long )kvaddr &PAGE_MASK ; +enum fixed_addresses idx; + +preempt_disable (); + idx = type +KM_TYPE_NR *smp_processor_id (); +if (vaddr ==__fix_to_virt (FIX_KMAP_BEGIN +idx)) { + current->kmap_atomic_ptes[idx] =native_make_pte (0 ); + current->kmap_atomic_count--; +kpte_clear_flush (kmap_pte - idx, vaddr); + }else { +#ifdef CONFIG_DEBUG_HIGHMEM +BUG_ON (vaddr <PAGE_OFFSET ); +BUG_ON (vaddr >= (unsigned long )high_memory); +#endif + } +arch_flush_lazy_mmu_mode (); +pagefault_enable (); +preempt_enable (); +} + +void *__kmap_atomic_pfn (unsigned long pfn,enum km_type type) +{ +enum fixed_addresses idx; +unsigned long vaddr; +pte_t pte; + +preempt_disable (); +pagefault_disable (); + + idx = type +KM_TYPE_NR *smp_processor_id (); + vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx); + pte =pfn_pte (pfn, kmap_prot); + + current->kmap_atomic_ptes[idx] = pte; + current->kmap_atomic_count++; + +set_pte (kmap_pte-idx, pte); +arch_flush_lazy_mmu_mode (); +preempt_enable (); + +return (void *) vaddr; +} + +void arch_kmap_atomic_save (void ) +{ +int i; + +if (!current->kmap_atomic_count) +return ; + +arch_enter_lazy_mmu_mode (); +for (i =0 ; i <KM_TYPE_NR ; i++) { +enum fixed_addresses idx = i +KM_TYPE_NR *smp_processor_id (); +unsigned long vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx); + +kpte_clear_flush (kmap_pte - idx, vaddr); + } +arch_leave_lazy_mmu_mode (); +} + +void arch_kmap_atomic_restore (void ) +{ +int i; + +if (!current->kmap_atomic_count) +return ; + +arch_enter_lazy_mmu_mode (); +for (i =0 ; i <KM_TYPE_NR ; i++) { +enum fixed_addresses idx = i +KM_TYPE_NR *smp_processor_id (); +pte_t pte = current->kmap_atomic_ptes[i]; + +if (!pte_none (pte)) +set_pte (kmap_pte - idx, pte); + } +arch_leave_lazy_mmu_mode (); +} + +#endif /* CONFIG_PREEMPT_RT */ + +void *__kmap_atomic (struct page *page,enum km_type type) +{ +return kmap_atomic_prot (page, type, kmap_prot); +} +struct page *__kmap_atomic_to_page (void *ptr) {unsigned long idx, vaddr = (unsigned long )ptr;
@@ -174,8 +290,6 @@ struct page *__kmap_atomic_to_page(void *ptr)return pte_page (*pte); } -EXPORT_SYMBOL (kmap); -EXPORT_SYMBOL (kunmap); -EXPORT_SYMBOL (kunmap_virt);EXPORT_SYMBOL (__kmap_atomic);EXPORT_SYMBOL (__kunmap_atomic); +
include/asm-x86/highmem.h +0 -11
@@ -84,22 +84,11 @@ struct page *kmap_atomic_to_page(void *ptr);#define flush_cache_kmaps () do { } while (0) -/* -* on PREEMPT_RT kmap_atomic() is a wrapper that uses kmap(): -*/ -#ifdef CONFIG_PREEMPT_RT -# define kmap_atomic_prot (page, type, prot) ({ pagefault_disable(); kmap(page); }) -# define kmap_atomic (page, type) ({ pagefault_disable(); kmap(page); }) -# define kmap_atomic_pfn (pfn, type) kmap(pfn_to_page(pfn)) -# define kunmap_atomic (kvaddr, type) do { pagefault_enable(); kunmap_virt(kvaddr); } while(0) -# define kmap_atomic_to_page (kvaddr) kmap_to_page(kvaddr) -#else # define kmap_atomic_prot (page, type, prot) __kmap_atomic_prot(page, type, prot)# define kmap_atomic (page, type) __kmap_atomic(page, type)# define kmap_atomic_pfn (pfn, type) __kmap_atomic_pfn(pfn, type)# define kunmap_atomic (kvaddr, type) __kunmap_atomic(kvaddr, type)# define kmap_atomic_to_page (kvaddr) __kmap_atomic_to_page(kvaddr) -#endif #endif /* __KERNEL__ */
include/linux/sched.h +5 -0
@@ -1441,6 +1441,11 @@ struct task_struct {*/ int in_printk;#endif + +#if defined(CONFIG_HIGHMEM ) && defined(CONFIG_PREEMPT_RT ) +int kmap_atomic_count; +pte_t kmap_atomic_ptes[KM_TYPE_NR ]; +#endif };#ifdef CONFIG_PREEMPT_RT
kernel/sched.c +10 -0
@@ -2623,6 +2623,14 @@ EXPORT_SYMBOL(preempt_enable_no_resched);#endif +#if defined(CONFIG_HIGHMEM ) && defined(CONFIG_PREEMPT_RT ) +extern void arch_kmap_atomic_save (void ); +extern void arch_kmap_atomic_restore (void ); +#else +static inline void arch_kmap_atomic_save (void ) { } +static inline void arch_kmap_atomic_restore (void ) { } +#endif +/** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch
@@ -2741,6 +2749,7 @@ context_switch(struct rq *rq, struct task_struct *prev, {struct mm_struct *mm, *oldmm; +arch_kmap_atomic_save ();prepare_task_switch (rq, prev, next);trace_sched_switch (rq, prev, next); mm = next->mm;
@@ -2788,6 +2797,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* frame will be invalid. */ finish_task_switch (this_rq (), prev); +arch_kmap_atomic_restore (); }/*
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative
kmap_atomicimplementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?
I'll give it a try. Anything in particular I should try to do? Or not do? Or watch for?
-- Fernando
Quoted 7 lines
--- arch/x86/mm/highmem_32.c | 130 ++++++++++++++++++++++++++++++++++++++++++--- include/asm-x86/highmem.h | 11 ---- include/linux/sched.h | 5 ++ kernel/sched.c | 10 ++++ 4 files changed, 137 insertions(+), 19 deletions(-)[MUNCH]
Peter Zijlstra
maintainerQuoted 10 lines · Fernando Lopez-Lezcano
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative
kmap_atomicimplementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?I'll give it a try. Anything in particular I should try to do? Or not do? Or watch for?
Lots of I/O should stress the i386 highmem stuff. If all is well it works, if not, crashes and splats.
Quoted 12 lines
-- Fernando
Quoted 7 lines
--- arch/x86/mm/highmem_32.c | 130 ++++++++++++++++++++++++++++++++++++++++++--- include/asm-x86/highmem.h | 11 ---- include/linux/sched.h | 5 ++ kernel/sched.c | 10 ++++ 4 files changed, 137 insertions(+), 19 deletions(-)[MUNCH]
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Fernando Lopez-Lezcano
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative
kmap_atomicimplementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?
Quoted 5 lines
Quoted 2 lines
I'll give it a try. Anything in particular I should try to do? Or not do? Or watch for?
Lots of I/O should stress the i386 highmem stuff. If all is well it works, if not, crashes and splats.
Crashes and splats on boot... sorry, I don't have a way to capture the kernel oops (or whatever happens, I can only see the end of the printout)
-- Fernando
PS: things left on the screen (top to bottom):
? speedstep_detect_processor
? __copy_from_user_ll_noccache_nozeroiov_iter_copy_from_user_atomic
? generic_file_buffered_writegeneric_file_buffered_write
? avs_has_perm_noaudit
? __rt_spin_lock
? selinux_inode_need_killpriv
? __rt_spin_lockmnt_drop_write__generic_file_aio_write_nolockgeneric_file_aio_writedo_sync_write
? __enqueue_entity
? autoremove_wake_function
? selinux_file_oermission
? security_file_permission
? do_sync_writevfs_writesys_write
? schedule_tail
? ret_from_fork
? thread_kernel_helper
Quoted 12 lines
Quoted 12 lines
-- Fernando
Quoted 7 lines
--- arch/x86/mm/highmem_32.c | 130 ++++++++++++++++++++++++++++++++++++++++++--- include/asm-x86/highmem.h | 11 ---- include/linux/sched.h | 5 ++ kernel/sched.c | 10 ++++ 4 files changed, 137 insertions(+), 19 deletions(-)[MUNCH]
Peter Zijlstra
maintainerQuoted 39 lines · Fernando Lopez-Lezcano
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Fernando Lopez-Lezcano
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative
kmap_atomicimplementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?Quoted 5 lines
Quoted 2 lines
I'll give it a try. Anything in particular I should try to do? Or not do? Or watch for?
Lots of I/O should stress the i386 highmem stuff. If all is well it works, if not, crashes and splats.
Crashes and splats on boot... sorry, I don't have a way to capture the kernel oops (or whatever happens, I can only see the end of the printout)
-- Fernando
PS: things left on the screen (top to bottom):
?
speedstep_detect_processor
?__copy_from_user_ll_noccache_nozeroiov_iter_copy_from_user_atomic
?generic_file_buffered_writegeneric_file_buffered_write
?avs_has_perm_noaudit
?__rt_spin_lock
?selinux_inode_need_killpriv
?__rt_spin_lockmnt_drop_write__generic_file_aio_write_nolockgeneric_file_aio_writedo_sync_write
?__enqueue_entity
?autoremove_wake_function
Quoted 8 lines
?
selinux_file_oermission
?security_file_permission
?do_sync_writevfs_writesys_write
?schedule_tail
?ret_from_fork
?thread_kernel_helper
Oh well, thanks for trying, I guess I need to go run this on real hardware, which means finding where I left this i386 distro on the test boxen :-)
Quoted 45 lines · Peter Zijlstra
Quoted 35 lines · Fernando Lopez-Lezcano
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Fernando Lopez-Lezcano
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
I've been working on an alternative
kmap_atomicimplementation for -rt, the below has been build and booted but not stressed, anybody care to give it a spin ?Quoted 5 lines
Quoted 2 lines
I'll give it a try. Anything in particular I should try to do? Or not do? Or watch for?
Lots of I/O should stress the i386 highmem stuff. If all is well it works, if not, crashes and splats.
Crashes and splats on boot... sorry, I don't have a way to capture the kernel oops (or whatever happens, I can only see the end of the printout)
?
speedstep_detect_processor
?__copy_from_user_ll_noccache_nozeroiov_iter_copy_from_user_atomic
?generic_file_buffered_writegeneric_file_buffered_write
?avs_has_perm_noaudit
?__rt_spin_lock
?selinux_inode_need_killpriv
?__rt_spin_lockmnt_drop_write__generic_file_aio_write_nolockgeneric_file_aio_writedo_sync_write
?__enqueue_entity
?autoremove_wake_functionQuoted 8 lines
?
selinux_file_oermission
?security_file_permission
?do_sync_writevfs_writesys_write
?schedule_tail
?ret_from_fork
?thread_kernel_helper
Quoted 3 lines
Oh well, thanks for trying, I guess I need to go run this on real hardware, which means finding where I left this i386 distro on the test boxen :-)
Let me know if you have something to test (or just maybe a simpler fix for the BUGs that I posted at the beginning of the thread?)
-- Fernando
Quoted 15 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
Quoted 1 line
As here.
Still happening in 2.6.26.8-rt13...
-- Fernando
Quoted 58 lines
Quoted 58 lines
Nov 7 10:31:19 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:19 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:19 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:19 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:19 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:19 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:19 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:19 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:19 host kernel: =======================Nov 7 10:31:01 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:01 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:01 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:01 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:01 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:31:01 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:31:01 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:31:01 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:31:01 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:31:01 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:31:01 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:31:01 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:31:01 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:31:01 host kernel: =======================Nov 7 10:30:58 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:30:58 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:30:58 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:30:58 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:30:58 host kernel: [<c041cd55>] kmap+0x42/0x55 Nov 7 10:30:58 host kernel: [<f88e9e1d>] ata_sff_hsm_move+0x3d3/0x66b [libata] Nov 7 10:30:58 host kernel: [<f88ea314>] ata_sff_interrupt+0x153/0x1eb [libata] Nov 7 10:30:58 host kernel: [<c0461730>] handle_IRQ_event+0x49/0xcd Nov 7 10:30:58 host kernel: [<c0461a15>] thread_simple_irq+0x45/0x7c Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c0461afb>] do_irqd+0xaf/0x229 Nov 7 10:30:58 host kernel: [<c0461a4c>] ? do_irqd+0x0/0x229 Nov 7 10:30:58 host kernel: [<c043b15f>] kthread+0x3b/0x61 Nov 7 10:30:58 host kernel: [<c043b124>] ? kthread+0x0/0x61 Nov 7 10:30:58 host kernel: [<c040581f>] kernel_thread_helper+0x7/0x10 Nov 7 10:30:58 host kernel: =======================
Steven Rostedt
reviewerQuoted 16 lines · Fernando Lopez-Lezcano
Quoted 15 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
Quoted 3 lines
Quoted 1 line
As here.
Still happening in 2.6.26.8-rt13...
Fernando,
Thanks for reporting this. Looks like we need to fix the ata driver:
2.6.26.8-rt13:
drivers/ata/libata-sff.c:
ata_sff_hsm_move()
calls ata_pio_sectors(), calls
ata_pio_sector()
/* FIXME: use a bounce buffer */ local_irq_save(flags); buf = kmap_atomic(page, KM_IRQ0);
This code is in 2.6.24-rt but in libata-core.c It is slightly different, prehaps you were just lucky?
-- Steve
Steven Rostedt
reviewerQuoted 16 lines · Fernando Lopez-Lezcano
Quoted 15 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
Quoted 3 lines
Quoted 1 line
As here.
Still happening in 2.6.26.8-rt13...
Could you try this patch and see if it fixes your issue.
Thanks,
-- Steve
--- drivers/ata/libata-sff.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/ata/libata-sff.c +6 -6
@@ -740,9 +740,9 @@ unsigned int ata_sff_data_xfer_noirq(strunsigned long flags;unsigned int consumed; -local_irq_save (flags); +local_irq_save_nort (flags); consumed =ata_sff_data_xfer (dev, buf, buflen, rw); -local_irq_restore (flags); +local_irq_restore_nort (flags);return consumed; }
@@ -780,7 +780,7 @@ static void ata_pio_sector(struct ata_quunsigned long flags;/* FIXME: use a bounce buffer */ -local_irq_save (flags); +local_irq_save_nort (flags); buf =kmap_atomic (page,KM_IRQ0 );/* do the actual data transfer */
@@ -788,7 +788,7 @@ static void ata_pio_sector(struct ata_qu do_write);kunmap_atomic (buf,KM_IRQ0 ); -local_irq_restore (flags); +local_irq_restore_nort (flags); }else { buf =page_address (page); ap->ops->sff_data_xfer(qc->dev, buf + offset, qc->sect_size,
@@ -918,14 +918,14 @@ next_sg:unsigned long flags;/* FIXME: use bounce buffer */ -local_irq_save (flags); +local_irq_save_nort (flags); buf =kmap_atomic (page,KM_IRQ0 );/* do the actual data transfer */ consumed = ap->ops->sff_data_xfer (dev, buf + offset, count, rw);kunmap_atomic (buf,KM_IRQ0 ); -local_irq_restore (flags); +local_irq_restore_nort (flags); }else { buf =page_address (page); consumed = ap->ops->sff_data_xfer (dev, buf + offset, count, rw);
Quoted 18 lines · Steven Rostedt
Quoted 16 lines · Fernando Lopez-Lezcano
Quoted 15 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
As here.
Still happening in 2.6.26.8-rt13...
Could you try this patch and see if it fixes your issue.
That particular BUG seems to be gone, so far with no side effects (no hard disk explosions or anything like that, hopefully). Thanks!!!
Hmmm, I don't remember seeing this before (but I could be wrong):
irq 19: nobody cared (try booting with the "irqpoll" option)
Pid: 479, comm: IRQ-19 Not tainted 2.6.26.8-1.rt13.2.fc9.ccrma.i686.rtPAE #1
[<c0461b70>] __report_bad_irq+0x2e/0x6f [<c0461d6c>] note_interrupt+0x1bb/0x20b [<c0460f6c>] ? handle_IRQ_event+0x49/0xe6 [<c046128c>] thread_simple_irq+0x68/0x7c [<c04612a0>] ? do_irqd+0x0/0x229 [<c046134f>] do_irqd+0xaf/0x229 [<c04612a0>] ? do_irqd+0x0/0x229 [<c043b023>] kthread+0x3b/0x61 [<c043afe8>] ? kthread+0x0/0x61 [<c040481f>] kernel_thread_helper+0x7/0x10 =======================
handlers:
[<c058d680>] (usb_hcd_irq+0x0/0x61)
turning off IO-APIC fast mode.
irq 19: nobody cared (try booting with the "irqpoll" option)
Pid: 479, comm: IRQ-19 Not tainted 2.6.26.8-1.rt13.2.fc9.ccrma.i686.rtPAE #1
[<c0461b70>] __report_bad_irq+0x2e/0x6f [<c0461d6c>] note_interrupt+0x1bb/0x20b [<c0460f6c>] ? handle_IRQ_event+0x49/0xe6 [<c046128c>] thread_simple_irq+0x68/0x7c [<c04612a0>] ? do_irqd+0x0/0x229 [<c046134f>] do_irqd+0xaf/0x229 [<c04612a0>] ? do_irqd+0x0/0x229 [<c043b023>] kthread+0x3b/0x61 [<c043afe8>] ? kthread+0x0/0x61 [<c040481f>] kernel_thread_helper+0x7/0x10 =======================
handlers:
[<c058d680>] (usb_hcd_irq+0x0/0x61)
Related?
-- Fernando
Quoted 55 lines
--- drivers/ata/libata-sff.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)Index:
linux-2.6.26.8-rt13/drivers/ata/libata-sff.c
===================================================================drivers/ata/libata-sff.c +6
-6@@ -740,9 +740,9 @@ unsigned int ata_sff_data_xfer_noirq(strunsigned long flags;unsigned int consumed; -local_irq_save (flags); +local_irq_save_nort (flags); consumed =ata_sff_data_xfer (dev, buf, buflen, rw); -local_irq_restore (flags); +local_irq_restore_nort (flags);return consumed; }@@ -780,7 +780,7 @@ static void ata_pio_sector(struct ata_quunsigned long flags;/* FIXME: use a bounce buffer */ -local_irq_save (flags); +local_irq_save_nort (flags); buf =kmap_atomic (page,KM_IRQ0 );/* do the actual data transfer */ @@ -788,7 +788,7 @@ static void ata_pio_sector(struct ata_qu do_write);kunmap_atomic (buf,KM_IRQ0 ); -local_irq_restore (flags); +local_irq_restore_nort (flags); }else { buf =page_address (page); ap->ops->sff_data_xfer(qc->dev, buf + offset, qc->sect_size,@@ -918,14 +918,14 @@ next_sg:unsigned long flags;/* FIXME: use bounce buffer */ -local_irq_save (flags); +local_irq_save_nort (flags); buf =kmap_atomic (page,KM_IRQ0 );/* do the actual data transfer */ consumed = ap->ops->sff_data_xfer (dev, buf + offset, count, rw);kunmap_atomic (buf,KM_IRQ0 ); -local_irq_restore (flags); +local_irq_restore_nort (flags); }else { buf =page_address (page); consumed = ap->ops->sff_data_xfer (dev, buf + offset, count, rw);
Steven Rostedt
reviewerQuoted 19 lines · Fernando Lopez-Lezcano
Quoted 18 lines · Steven Rostedt
Quoted 16 lines · Fernando Lopez-Lezcano
Quoted 15 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
As here.
Still happening in 2.6.26.8-rt13...
Could you try this patch and see if it fixes your issue.
That particular BUG seems to be gone, so far with no side effects (no hard disk explosions or anything like that, hopefully). Thanks!!!
Great to hear!
Quoted 34 lines
Hmmm, I don't remember seeing this before (but I could be wrong):
irq 19: nobody cared (try booting with the "irqpoll" option)
Pid: 479, comm: IRQ-19 Not tainted 2.6.26.8-1.rt13.2.fc9.ccrma.i686.rtPAE #1[<c0461b70>] __report_bad_irq+0x2e/0x6f [<c0461d6c>] note_interrupt+0x1bb/0x20b [<c0460f6c>] ? handle_IRQ_event+0x49/0xe6 [<c046128c>] thread_simple_irq+0x68/0x7c [<c04612a0>] ? do_irqd+0x0/0x229 [<c046134f>] do_irqd+0xaf/0x229 [<c04612a0>] ? do_irqd+0x0/0x229 [<c043b023>] kthread+0x3b/0x61 [<c043afe8>] ? kthread+0x0/0x61 [<c040481f>] kernel_thread_helper+0x7/0x10 =======================handlers:
[<c058d680>] (usb_hcd_irq+0x0/0x61)
turning off IO-APIC fast mode.
irq 19: nobody cared (try booting with the "irqpoll" option)
Pid: 479, comm: IRQ-19 Not tainted 2.6.26.8-1.rt13.2.fc9.ccrma.i686.rtPAE #1[<c0461b70>] __report_bad_irq+0x2e/0x6f [<c0461d6c>] note_interrupt+0x1bb/0x20b [<c0460f6c>] ? handle_IRQ_event+0x49/0xe6 [<c046128c>] thread_simple_irq+0x68/0x7c [<c04612a0>] ? do_irqd+0x0/0x229 [<c046134f>] do_irqd+0xaf/0x229 [<c04612a0>] ? do_irqd+0x0/0x229 [<c043b023>] kthread+0x3b/0x61 [<c043afe8>] ? kthread+0x0/0x61 [<c040481f>] kernel_thread_helper+0x7/0x10 =======================
Quoted 4 lines
handlers:
[<c058d680>] (usb_hcd_irq+0x0/0x61)Related?
No, but that looks like a boot interrupt issue.
See this (huge) thread for details:
http://marc.info/?t=123154237900002&r=1&w=2
-- Steve
Quoted 23 lines · Steven Rostedt
Quoted 21 lines · Fernando Lopez-Lezcano
Quoted 20 lines · Steven Rostedt
Quoted 18 lines · Fernando Lopez-Lezcano
Quoted 16 lines · Steven Rostedt
Peter, I think we've seen this before. It is the highmem code sleeping.
Quoted 10 lines · Fernando Lopez-Lezcano
Three BUGs while booting 2.6.26.7 + rt11 (same hardware: no problems with 2.6.24.7-rt21):
Nov 7 10:31:19 host kernel: BUG: sleeping function called from invalid context IRQ-22(540) at arch/x86/mm/highmem_32.c:8 Nov 7 10:31:19 host kernel: in_atomic():0 [00000000], irqs_disabled():1 Nov 7 10:31:19 host kernel: Pid: 540, comm: IRQ-22 Not tainted 2.6.26.7-1.rt11.1fc9.ccrma.i686.rt #1 Nov 7 10:31:19 host kernel: [<c041ff88>] __might_sleep+0xe8/0xed Nov 7 10:31:19 host kernel: [<c041cd55>] kmap+0x42/0x55As here.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Still happening in 2.6.26.8-rt13...
Could you try this patch and see if it fixes your issue.
That particular BUG seems to be gone, so far with no side effects (no hard disk explosions or anything like that, hopefully). Thanks!!!
Great to hear!
Hmmm, it is not happening on my laptop but on different hardware I see this:
Uniform CD-ROM driver Revision: 3.20
sr 2:0:0:0: Attached scsi CD-ROM sr0
BUG: sleeping function called from invalid context IRQ-14(959) at
arch/x86/mm/highmem_32.c:8
in_atomic():0 [00000000], irqs_disabled():1
Pid: 959, comm: IRQ-14 Not tainted 2.6.26.8-1.rt13.2.fc9.ccrma.i686.rt #1
[<c041ff73>] __might_sleep+0xe8/0xed [<c041cd55>] kmap+0x42/0x55 [<c0504830>] sg_copy_buffer+0xa6/0x16c [<c0504903>] sg_copy_to_buffer+0xd/0xf [<f88cd46f>] atapi_qc_complete+0x25c/0x2b6 [libata] [<f88c75e5>] __ata_qc_complete+0x8e/0x93 [libata] [<f88c8241>] ata_qc_complete+0x197/0x19d [libata] [<f88d3a01>] ata_hsm_qc_complete+0x9b/0xb3 [libata] [<f88d401b>] ata_sff_hsm_move+0x602/0x64e [libata] [<c04248d4>] ? hrtick_set+0x80/0xe5 [<c0643921>] ? __rt_spin_lock+0x24/0x61 [<f88d42c6>] ata_sff_interrupt+0x153/0x1eb [libata] [<c0460fbc>] handle_IRQ_event+0x49/0xe6 [<c046141b>] do_irqd+0x12b/0x229 [<c04612f0>] ? do_irqd+0x0/0x229 [<c043b073>] kthread+0x3b/0x61 [<c043b038>] ? kthread+0x0/0x61 [<c040581f>] kernel_thread_helper+0x7/0x10 =======================
ata6: SATA link down (SStatus 0 SControl 310)
ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
ACPI: PCI Interrupt 0000:02:0e.0[A] -> Link [APC3] -> GSI 18 (level, low) -> IRQ 18
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[18] MMIO=[f5118000-f51187ff] Max Packet=[4096] IR/IT contexts=[ 4/8]
-- Fernando