"kernel BUG at mm/rmap.c:474!" error on 2.6.9
Subsystems (3)
What could be the real cause of such messages? (I assume that it was not really apache or php fault (httpd and php shown in logs below)).
Kernel 2.6.9, SATA drive on intel chipset (ICH5), processor is P4 with HT, 1GB of memory. Glibc 2.3.2, gcc version 3.3.4.
On console I've got:
Message from syslogd@sh at Tue Jan 11 10:01:01 2005 ...
sh kernel: Bad page state at prep_new_page (in process 'php', page c17b4580)
Message from syslogd@sh at Tue Jan 11 10:01:01 2005 ... sh kernel: flags:0x40000114 mapping:00000000 mapcount:-1 count:0
Message from syslogd@sh at Tue Jan 11 10:01:01 2005 ... sh kernel: Backtrace:
Message from syslogd@sh at Tue Jan 11 10:01:01 2005 ... sh kernel: Trying to fix it up, but a reboot is needed
Quoted 1 line
From logs:
Jan 11 10:00:44 sh kernel: ------------[ cut here ]------------ Jan 11 10:00:44 sh kernel: kernel BUG at mm/rmap.c:474! Jan 11 10:00:44 sh kernel: invalid operand: 0000 [#1] Jan 11 10:00:44 sh kernel: PREEMPT SMP Jan 11 10:00:44 sh kernel: Modules linked in: i8xx_tco ide_scsi e1000 Jan 11 10:00:44 sh kernel: CPU: 0 Jan 11 10:00:44 sh kernel: EIP: 0060:[<c014ea30>] Not tainted VLI Jan 11 10:00:44 sh kernel: EFLAGS: 00010286 (2.6.9) Jan 11 10:00:44 sh kernel: EIP is at page_remove_rmap+0x3f/0x53 Jan 11 10:00:44 sh kernel: eax: ffffffff ebx: 00000000 ecx: c17b4580 edx: c17b4580 Jan 11 10:00:44 sh kernel: esi: de3dd27c edi: c17b4580 ebp: 00000020 esp: ce236e84 Jan 11 10:00:44 sh kernel: ds: 007b es: 007b ss: 0068 Jan 11 10:00:44 sh kernel: Process httpd (pid: 13984, threadinfo=ce236000 task=d01ff730) Jan 11 10:00:44 sh kernel: Stack: c01485b5 c17b4580 00000000 e4efcf3c c0125ffc b7c9f000 d6773b7c b78a0000 Jan 11 10:00:44 sh kernel: 00000000 c0148789 c180f400 d6773b78 b789f000 00001000 00000000 c180f400 Jan 11 10:00:44 sh kernel: b789f000 d6773b7c b78a0000 00000000 c01487f0 c180f400 d6773b78 b789f000 Jan 11 10:00:44 sh kernel: Call Trace: Jan 11 10:00:44 sh kernel: [<c01485b5>] zap_pte_range+0x13f/0x2bc Jan 11 10:00:44 sh kernel: [<c0125ffc>] update_process_times+0x45/0x51 Jan 11 10:00:44 sh kernel: [<c0148789>] zap_pmd_range+0x57/0x73 Jan 11 10:00:44 sh kernel: [<c01487f0>] unmap_page_range+0x4b/0x71 Jan 11 10:00:44 sh kernel: [<c0148917>] unmap_vmas+0x101/0x20e Jan 11 10:00:44 sh kernel: [<c014cd36>] exit_mmap+0xa5/0x187 Jan 11 10:00:44 sh kernel: [<c011b93b>] mmput+0x66/0x91 Jan 11 10:00:44 sh kernel: [<c0120015>] do_exit+0x17b/0x42a Jan 11 10:00:44 sh kernel: [<c0120370>] do_group_exit+0x3c/0xa7 Jan 11 10:00:44 sh kernel: [<c01050fb>] syscall_call+0x7/0xb Jan 11 10:00:44 sh kernel: Code: c0 74 27 8b 42 08 83 c0 01 78 20 9c 59 fa b8 00 f0 ff ff ba 20 7c 47 c0 21 e0 8b 40 10 03 14 85 20 b0 47 c0 83 6a 10 01 51 9d c3 <0f> 0b da 01 1d 31 36 c0 eb d6 0f 0b d7 01 1d 31 36 c0 eb b8 83 Jan 11 10:00:44 sh kernel: <6>note: httpd[13984] exited with preempt_count 1 Jan 11 10:01:01 sh kernel: Bad page state at prep_new_page (in process 'php', page c17b4580) Jan 11 10:01:01 sh kernel: flags:0x40000114 mapping:00000000 mapcount:-1 count:0 Jan 11 10:01:01 sh kernel: Backtrace: Jan 11 10:01:01 sh kernel: [<c013e547>] bad_page+0x78/0xa4 Jan 11 10:01:01 sh kernel: [<c013e8a6>] prep_new_page+0x32/0x51 Jan 11 10:01:01 sh kernel: [<c013edf4>] buffered_rmqueue+0x10f/0x1fa Jan 11 10:01:01 sh kernel: [<c013f216>] __alloc_pages+0x337/0x358 Jan 11 10:01:01 sh kernel: [<c0149c6e>] do_anonymous_page+0x98/0x1a5 Jan 11 10:01:01 sh kernel: [<c0149dde>] do_no_page+0x63/0x324 Jan 11 10:01:01 sh kernel: [<c014a29b>] handle_mm_fault+0xf4/0x16c Jan 11 10:01:01 sh kernel: [<c011713e>] do_page_fault+0x39d/0x5a4 Jan 11 10:01:01 sh kernel: [<c01cfaba>] journal_stop+0x1ad/0x26e Jan 11 10:01:01 sh kernel: [<c01c4224>] ext3_mark_iloc_dirty+0x29/0x37 Jan 11 10:01:01 sh kernel: [<c01c434d>] ext3_mark_inode_dirty+0x4f/0x51 Jan 11 10:01:01 sh kernel: [<c01c862d>] __ext3_journal_stop+0x24/0x50 Jan 11 10:01:01 sh kernel: [<c01c43b8>] ext3_dirty_inode+0x69/0x87 Jan 11 10:01:01 sh kernel: [<c0116da1>] do_page_fault+0x0/0x5a4 Jan 11 10:01:01 sh kernel: [<c0105b85>] error_code+0x2d/0x38 Jan 11 10:01:01 sh kernel: [<c013b475>] file_read_actor+0x20/0xec Jan 11 10:01:01 sh kernel: [<c013b0bc>] do_generic_mapping_read+0x1a6/0x53f Jan 11 10:01:01 sh kernel: [<c013b735>] __generic_file_aio_read+0x1f4/0x226 Jan 11 10:01:01 sh kernel: [<c013b455>] file_read_actor+0x0/0xec Jan 11 10:01:01 sh kernel: [<c016f2a0>] dput+0x24/0x1af Jan 11 10:01:01 sh kernel: [<c013b7c1>] generic_file_aio_read+0x5a/0x74 Jan 11 10:01:01 sh kernel: [<c01580de>] do_sync_read+0xbe/0xeb Jan 11 10:01:01 sh kernel: [<c014bced>] do_mmap_pgoff+0x44e/0x735 Jan 11 10:01:01 sh kernel: [<c011b636>] autoremove_wake_function+0x0/0x57 Jan 11 10:01:01 sh kernel: [<c01581bb>] vfs_read+0xb0/0x119 Jan 11 10:01:01 sh kernel: [<c0158479>] sys_read+0x51/0x80 Jan 11 10:01:01 sh kernel: [<c01050fb>] syscall_call+0x7/0xb Jan 11 10:01:01 sh kernel: Trying to fix it up, but a reboot is needed
And after ~15 minutes machine rebooted (possibly it was caused by i8xx_tco module and softdog).
Signature · 1 line
Daniel Fenert --==> [email protected] <==--
Quoted 2 lines · Daniel Fenert
What could be the real cause of such messages? (I assume that it was not really apache or php fault (httpd and php shown in logs below)).
We still do not know; we'd very much like to know.
It would not be the fault of any userspace program (unless they corrupt via /dev/mem or something like that).
It may be a core kernel problem, but I've searched repeatedly and failed. It may be a driver problem e.g. GregKH's incident suggested a problem in DRM, and Andrea has pointed to a worrying ioctl there (looks like it could ClearPageReserved too early): I've been halfway through following that up for a few weeks now. Are you using DRM? (but the hallmarks in your case are different.)
It can be caused by somewhere freeing a page it no longer holds; but in that case we'd usually expect to see the Bad page state error coming from free_pages_check rather than prep_new_page, and to be followed by the rmap.c BUG rather than following it.
It could easily be caused by bad memory bitflipping in a page table (but in general, we'd expect to be hearing of swap_free errors, or random corruption, if that were generally the case - I think). Please give memtest86 a good run to rule out that possibility.
If memtest86 is satifisfied, would you mind running with the patch below (against 2.6.9, suitable for i386 or x86_64, but not suitable for the various architectures which use PG_arch_1)? To give us more debug info - it's unlikely to solve the mystery on it's own, but I hope it might help us to look in the right direction. And send me any "Bad rmap" and "Bad page state" log entries you find (but perhaps this was a one-off, and nothing more will appear).
Thanks,
Hugh
Quoted 9 lines
Kernel 2.6.9, SATA drive on intel chipset (ICH5), processor is P4 with HT, 1GB of memory. Glibc 2.3.2, gcc version 3.3.4.
Jan 11 10:00:44 sh kernel: kernel BUG at mm/rmap.c:474! Jan 11 10:00:44 sh kernel: EIP is at page_remove_rmap+0x3f/0x53 Jan 11 10:00:44 sh kernel: eax: ffffffff ebx: 00000000 ecx: c17b4580 edx: c17b4580 Jan 11 10:00:44 sh kernel: esi: de3dd27c edi: c17b4580 ebp: 00000020 esp: ce236e84 Jan 11 10:01:01 sh kernel: Bad page state at prep_new_page (in process 'php', page c17b4580) Jan 11 10:01:01 sh kernel: flags:0x40000114 mapping:00000000 mapcount:-1 count:0
include/linux/rmap.h +1 -1
@@ -71,7 +71,7 @@ void __anon_vma_link(struct vm_area_stru*/ void page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long );void page_add_file_rmap (struct page *); -void page_remove_rmap (struct page *); +void page_remove_rmap (struct page *,struct vm_area_struct *,unsigned long );/** * page_dup_rmap - duplicate pte mapping to a page
mm/fremap.c +1 -1
@@ -36,7 +36,7 @@ static inline void zap_pte(struct mm_strif (!PageReserved (page)) {if (pte_dirty (pte))set_page_dirty (page); -page_remove_rmap (page); +page_remove_rmap (page, vma, addr);page_cache_release (page); mm->rss--; }
mm/memory.c +6 -4
@@ -356,6 +356,7 @@ nomem: }static void zap_pte_range (struct mmu_gather *tlb, +struct vm_area_struct *vma,pmd_t *pmd,unsigned long address,unsigned long size,struct zap_details *details) {
@@ -419,7 +420,7 @@ static void zap_pte_range(struct mmu_gatif (pte_young (pte) && !PageAnon (page))mark_page_accessed (page); tlb->freed++; -page_remove_rmap (page); +page_remove_rmap (page, vma, address+offset);tlb_remove_page (tlb, page);continue ; }
@@ -437,6 +438,7 @@ static void zap_pte_range(struct mmu_gat }static void zap_pmd_range (struct mmu_gather *tlb, +struct vm_area_struct *vma,pgd_t * dir,unsigned long address,unsigned long size,struct zap_details *details) {
@@ -455,7 +457,7 @@ static void zap_pmd_range(struct mmu_gatif (end > ((address +PGDIR_SIZE ) &PGDIR_MASK )) end = ((address +PGDIR_SIZE ) &PGDIR_MASK );do { -zap_pte_range (tlb, pmd, address, end - address, details); +zap_pte_range (tlb, vma, pmd, address, end - address, details); address = (address +PMD_SIZE ) &PMD_MASK ; pmd++; }while (address && (address < end));
@@ -471,7 +473,7 @@ static void unmap_page_range(struct mmu_ dir =pgd_offset (vma->vm_mm, address);tlb_start_vma (tlb, vma);do { -zap_pmd_range (tlb, dir, address, end - address, details); +zap_pmd_range (tlb, vma, dir, address, end - address, details); address = (address +PGDIR_SIZE ) &PGDIR_MASK ; dir++; }while (address && (address < end));
@@ -1098,7 +1100,7 @@ static int do_wp_page(struct mm_struct *if (PageReserved (old_page)) ++mm->rss;else -page_remove_rmap (old_page); +page_remove_rmap (old_page, vma, address);break_cow (vma, new_page, address, page_table);lru_cache_add_active (new_page);page_add_anon_rmap (new_page, vma, address);
mm/rmap.c +33 -5
@@ -453,8 +453,12 @@ void page_add_anon_rmap(struct page *pagvoid page_add_file_rmap (struct page *page) {BUG_ON (PageAnon (page)); -if (!pfn_valid (page_to_pfn (page)) ||PageReserved (page)) +if (!pfn_valid (page_to_pfn (page)))return ; +if (PageReserved (page)) { +set_bit (PG_arch_1, &page->flags); +return ; + }if (atomic_inc_and_test (&page->_mapcount))inc_page_state (nr_mapped);
@@ -466,12 +470,36 @@ void page_add_file_rmap(struct page *pag* * Caller needs to hold the mm->page_table_lock. */ -void page_remove_rmap (struct page *page) +void page_remove_rmap (struct page *page, +struct vm_area_struct *vma,unsigned long address) { +struct address_space *mapping =NULL ; +unsigned long index; +BUG_ON (PageReserved (page)); + index = (address - vma->vm_start) >>PAGE_SHIFT ; + index += vma->vm_pgoff; + +if (PageAnon (page)) + mapping = (void *) vma->anon_vma +PAGE_MAPPING_ANON ; +else if (!(vma->vm_flags & (VM_IO |VM_RESERVED ))) + mapping = vma->vm_file? vma->vm_file->f_mapping: (void *)(-1 ); + +if (page_mapcount (page) <=0 ||page_count (page) <=0 || + (mapping && (mapping != page->mapping || index != page->index)) || +test_bit (PG_arch_1, &page->flags)) { +printk ("Bad rmap in %s: page %p flags %lx " +"count %d mapcount %d addr %lx vm_flags %lx\n" , + current->comm, page, (unsigned long )page->flags, +page_count (page),page_mapcount (page), address, + vma->vm_flags); +printk ("\tpage mapping %p index %lx vma mapping %p index %lx\n" , + page->mapping, page->index, mapping, index); +get_page (page); +return ; + }if (atomic_add_negative (-1 , &page->_mapcount)) { -BUG_ON (page_mapcount (page) <0 );/* * It would be tidy to reset the PageAnon mapping here, * but that might overwrite a racing page_add_anon_rmap
@@ -581,7 +609,7 @@ static int try_to_unmap_one(struct page } mm->rss--; -page_remove_rmap (page); +page_remove_rmap (page, vma, address);page_cache_release (page); out_unmap:
@@ -678,7 +706,7 @@ static void try_to_unmap_cluster(unsigneif (pte_dirty (pteval))set_page_dirty (page); -page_remove_rmap (page); +page_remove_rmap (page, vma, address);page_cache_release (page); mm->rss--; (*mapcount)--;
W dniu Tue, Jan 11, 2005 at 02:09:51PM +0000, Hugh Dickins wystukał(a):
Quoted 7 lines
Quoted 2 lines
What could be the real cause of such messages? (I assume that it was not really apache or php fault (httpd and php shown in logs below)).
It could easily be caused by bad memory bitflipping in a page table (but in general, we'd expect to be hearing of
swap_freeerrors, or random corruption, if that were generally the case - I think). Please give memtest86 a good run to rule out that possibility.
I hardly can touch this machine, but I'll try tomorrow morning run memtest86 for some 20 minutes.
Quoted 7 lines
If memtest86 is satifisfied, would you mind running with the patch below (against 2.6.9, suitable for i386 or
x86_64, but not suitable for the various architectures which usePG_arch_1)? To give us more debug info - it's unlikely to solve the mystery on it's own, but I hope it might help us to look in the right direction. And send me any "Bad rmap" and "Bad page state" log entries you find (but perhaps this was a one-off, and nothing more will appear).
I'll try the patch.
This has happend for the first time (I have some problems with this machine, but utill now there were only httpd(php) processes eating all memory until machine is locked - that's why software watchdog is running there).
Signature · 4 lines
Daniel Fenert --==> [email protected] <==-- ====-P o w e r e d--b y--S l a c k w a r e-===-ICQ #37739641-==== Wonderful day. Your hangover just makes it seem terrible. ==========- http://daniel.fenert.net/ -==========< +48604628083 >
Quoted 2 lines · Daniel Fenert
I hardly can touch this machine,
Understood: thanks for trying.
Quoted 1 line
but I'll try tomorrow morning run memtest86 for some 20 minutes.
If it reports an error, that will be significant. But if it does not, I don't think we can conclude anything from such a short run. Anyway, better than nothing.
Quoted 1 line
I'll try the patch.
Much appreciated.
Hugh