Bad page state on AMD Opteron Dual System with kernel 2.6.13-rc6-git13
Hi there,
i have read some postings concerning the following Kernel Messages:
Aug 26 18:04:01 montdsnsu3 kernel: grep[11619] general protection rip:2aaaaaaaed43 rsp:7fffff9c0740 error:0 Aug 26 18:08:02 montdsnsu3 kernel: ping[14867] general protection rip:2aaaaaaaed43 rsp:7fffffdbf300 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[14987] general protection rip:2aaaaaaaed43 rsp:7fffffdbfce0 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[15041] general protection
rip:2aaaaaaaed43 rsp:7fffff9bf550 error:0
And the Bad Page State Messages:
Bad page state at prep_new_page (in process 'sh', page ffff8100011a69c8) flags:0x0100000000000014 mapping:0000000000000000 mapcount:-3 count:0
Backtrace:
Call Trace:<ffffffff8015a653>{bad_page+99}
<ffffffff8015aa31>{prep_new_page+65}
<ffffffff8015b1e2>{buffered_rmqueue+306}
<ffffffff8015b425>{__alloc_pages+261}
<ffffffff8015b7c5>{get_zeroed_page+37}
<ffffffff801691b7>{__pmd_alloc+55}
<ffffffff8016614e>{copy_page_range+462}
<ffffffff80131da4>{copy_mm+820}
<ffffffff80132cba>{copy_process+2282}
<ffffffff80133407>{do_fork+215}
<ffffffff8010db7a>{system_call+126}<ffffffff8010deeb>{ptregscall_common+103}
Trying to fix it up, but a reboot is needed
I have the same issues on an SUN V20z with an dual opteron 248.
montdsnsu3:~# lspci 0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:01:05.0 VGA compatible controller: Trident Microsystems Blade 3D PCI/AGP (rev 3a) 0000:02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:02:02.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:02:04.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
With the running kernel i get 2 kernel panics within the last week and the machine crash totally.
I would like to offer my help if i can do anything in debugging this or deal with more informations to fix this issue.
HTH,
weiti
Signature · 6 lines
Interpunktion und Orthographie dieser Email ist frei erfunden. Eine Übereinstimmung mit aktuellen oder ehemaligen Regeln wäre rein zufällig und ist nicht beabsichtigt. Tim Weippert <[email protected]> http://www.topf-sicret.org/
Hi,
Quoted 28 lines · Tim Weippert
i have read some postings concerning the following Kernel Messages:
Aug 26 18:04:01 montdsnsu3 kernel: grep[11619] general protection rip:2aaaaaaaed43 rsp:7fffff9c0740 error:0 Aug 26 18:08:02 montdsnsu3 kernel: ping[14867] general protection rip:2aaaaaaaed43 rsp:7fffffdbf300 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[14987] general protection rip:2aaaaaaaed43 rsp:7fffffdbfce0 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[15041] general protectionrip:2aaaaaaaed43 rsp:7fffff9bf550 error:0
And the Bad Page State Messages:
Bad page state at
prep_new_page(in process 'sh', pageffff8100011a69c8) flags:0x0100000000000014 mapping:0000000000000000 mapcount:-3 count:0Backtrace: Call Trace:<ffffffff8015a653>{bad_page+99} <ffffffff8015aa31>{prep_new_page+65} <ffffffff8015b1e2>{buffered_rmqueue+306} <ffffffff8015b425>{__alloc_pages+261} <ffffffff8015b7c5>{get_zeroed_page+37} <ffffffff801691b7>{__pmd_alloc+55} <ffffffff8016614e>{copy_page_range+462} <ffffffff80131da4>{copy_mm+820} <ffffffff80132cba>{copy_process+2282} <ffffffff80133407>{do_fork+215} <ffffffff8010db7a>{system_call+126}
Quoted 3 lines
<ffffffff8010deeb>{ptregscall_common+103}
Trying to fix it up, but a reboot is needed
Seems to be an identical problem as was filed here:
http://bugs.gentoo.org/show_bug.cgi?id=103497
This bug report seems to suggest that the ondemand scaling governor may be at fault. Does your setup use this too?
(CC'ing some extra people to make sure problem is known)
Quoted 51 lines
I have the same issues on an SUN V20z with an dual opteron 248.
montdsnsu3:~# lspci 0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge 0000:01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:01:05.0 VGA compatible controller: Trident Microsystems Blade 3D PCI/AGP (rev 3a) 0000:02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:02:02.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:02:04.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)With the running kernel i get 2 kernel panics within the last week and the machine crash totally.
I would like to offer my help if i can do anything in debugging this or deal with more informations to fix this issue.
HTH,
weiti
Quoted 35 lines · Daniel Drake
Hi,
Quoted 28 lines · Tim Weippert
i have read some postings concerning the following Kernel Messages:
Aug 26 18:04:01 montdsnsu3 kernel: grep[11619] general protection rip:2aaaaaaaed43 rsp:7fffff9c0740 error:0 Aug 26 18:08:02 montdsnsu3 kernel: ping[14867] general protection rip:2aaaaaaaed43 rsp:7fffffdbf300 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[14987] general protection rip:2aaaaaaaed43 rsp:7fffffdbfce0 error:0 Aug 26 18:08:03 montdsnsu3 kernel: grep[15041] general protectionrip:2aaaaaaaed43 rsp:7fffff9bf550 error:0
And the Bad Page State Messages:
Bad page state at
prep_new_page(in process 'sh', pageffff8100011a69c8) flags:0x0100000000000014 mapping:0000000000000000 mapcount:-3 count:0Backtrace: Call Trace:<ffffffff8015a653>{bad_page+99} <ffffffff8015aa31>{prep_new_page+65} <ffffffff8015b1e2>{buffered_rmqueue+306} <ffffffff8015b425>{__alloc_pages+261} <ffffffff8015b7c5>{get_zeroed_page+37} <ffffffff801691b7>{__pmd_alloc+55} <ffffffff8016614e>{copy_page_range+462} <ffffffff80131da4>{copy_mm+820} <ffffffff80132cba>{copy_process+2282} <ffffffff80133407>{do_fork+215} <ffffffff8010db7a>{system_call+126}Quoted 3 lines
<ffffffff8010deeb>{ptregscall_common+103}
Trying to fix it up, but a reboot is needed
Quoted 9 lines
Seems to be an identical problem as was filed here:
http://bugs.gentoo.org/show_bug.cgi?id=103497This bug report seems to suggest that the ondemand scaling governor may be at fault. Does your setup use this too?
(CC'ing some extra people to make sure problem is known)
As this is an Server, i don't even use cpufreq on this machine. So it think this isn't the same problem ...
Kind regards,
weiti
p.s: Please CC me, cause i am not subscribed on lkml.
Signature · 6 lines
Interpunktion und Orthographie dieser Email ist frei erfunden. Eine Übereinstimmung mit aktuellen oder ehemaligen Regeln wäre rein zufällig und ist nicht beabsichtigt. Tim Weippert <[email protected]> http://www.topf-sicret.org/
Hi,
Quoted 7 lines · Tim Weippert
Quoted 5 lines · Daniel Drake
Seems to be an identical problem as was filed here:
http://bugs.gentoo.org/show_bug.cgi?id=103497
Quoted 8 lines
Quoted 5 lines
This bug report seems to suggest that the ondemand scaling governor may be at fault. Does your setup use this too?
(CC'ing some extra people to make sure problem is known)
As this is an Server, i don't even use cpufreq on this machine. So it think this isn't the same problem ...
Update, with stable 2.6.13. I get nearly the same behavior.
One new oops:
swap_free: Bad swap file entry c000007fffff802fswap_free: Bad swap file entry c800007fffff802fswap_free: Bad swap file entry d000007fffff802fswap_free: Bad swap file entry d800007fffff802fswap_free: Bad swap file entry e000007fffff802fswap_free: Bad swap file entry 4000000000000000 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at "mm/rmap.c":493
invalid operand: 0000 [1] SMP
CPU 1
Modules linked in: autofs4 floppy i2c_amd756 i2c_core hw_random ohci_hcd
tg3 tsdev evdev evbug psmouse genrtc unix
Pid: 9014, comm: sh Not tainted 2.6.13
RIP: 0010:[<ffffffff8016e9ab>] <ffffffff8016e9ab>{page_remove_rmap+43}
RSP: 0018:ffff8100481c3da0 EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff81004a5fc420 RCX: ffff81000000d000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8100011a69c8
RBP: 0000000000484000 R08: 0000000000000001 R09: 000000000000000f
R10: 0000000000000001 R11: 0000000000000000 R12: 00000000078bfbff
R13: ffff810040e133e0 R14: ffff8100011a69c8 R15: 0000000000000000
FS: 00000000457ff970(0000) GS:ffffffff8056f880(0000)knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaaabd000 CR3: 0000000048205000 CR4: 00000000000006e0 Process sh (pid: 9014, threadinfo ffff8100481c2000, task ffff810048e7e270)
Stack: ffffffff801663f4 0000000000497000 ffff81004937f010
0000000000497000
0000000000497000 0000000000496fff ffff8100497dd000
0000000000497000
ffffffff801666ab 0000000000000000
Call Trace:<ffffffff801663f4>{zap_pte_range+436}
<ffffffff801666ab>{unmap_page_range+507}
<ffffffff80166815>{unmap_vmas+293}
<ffffffff8016c4d2>{exit_mmap+162}
<ffffffff801318b1>{mmput+49} <ffffffff80136d3a>{do_exit+442}
<ffffffff801370c0>{sys_exit_group+0}<ffffffff8010db7a>{system_call+126}
Code: 0f 0b a3 b4 5b 3f 80 ff ff ff ff c2 ed 01 66 66 66 90 66 66
RIP <ffffffff8016e9ab>{page_remove_rmap+43} RSP <ffff8100481c3da0>
<1>Fixing recursive fault but reboot is needed!
With this i get an hanging [sh] process which can't be killed, only cleanable with reboot:
www-data 7701 0.0 0.3 74448 6452 ? S 11:56 0:00/usr/sbin/cactid 0 93
www-data 7721 0.0 0.5 56296 10504 ? S 11:56 0:00 \_/usr/bin/php /usr/share/cacti/site/script_server.php cactid 0
www-data 9014 0.0 0.0 0 0 ? D 11:56 0:00 \_ [sh]
The machine is an cacti system with generally high load ... seems the kernel does only have problems on higher load.
HTH,
weiti
Signature · 6 lines
Interpunktion und Orthographie dieser Email ist frei erfunden. Eine Übereinstimmung mit aktuellen oder ehemaligen Regeln wäre rein zufällig und ist nicht beabsichtigt. Tim Weippert <[email protected]> http://www.topf-sicret.org/
Quoted 2 lines · Tim Weippert
As this is an Server, i don't even use cpufreq on this machine. So it think this isn't the same problem ...
Maybe you could post your experiences here:
http://bugzilla.kernel.org/show_bug.cgi?id=5133
Daniel
Quoted 2 lines · you
Hi,
8<
Quoted 3 lines
Update, with stable 2.6.13. I get nearly the same behavior.
I haven't tried 2.6.13 yet (still downloading), could you first try this (with yor last working kernel, since you seem to have a problem with 2.6.13)
echo 0 > /proc/sys/kernel/randomize_va_space
If this "hides" the problems for you, please take a look at this bug report, and add your details:
http://bugzilla.kernel.org/show_bug.cgi?id=4851
Regards
Quoted 4 lines · Bongani Hlope
Quoted 2 lines · you
Hi,
Quoted 8 lines
8<
Quoted 3 lines
Update, with stable 2.6.13. I get nearly the same behavior.
I haven't tried 2.6.13 yet (still downloading), could you first try this (with yor last working kernel, since you seem to have a problem with 2.6.13)
It's not only with 2.6.13, i think it is with all kernels > 2.6.9 or .10.
Quoted 1 line
echo 0 >
/proc/sys/kernel/randomize_va_space
Ok, i have set this within my 2.6.13 kernel and will look what happens.
Quoted 4 lines
If this "hides" the problems for you, please take a look at this bug report, and add your details:
Ok, i will add my details within the next days after examine if this helps.
bye,
tim
Signature · 6 lines
Interpunktion und Orthographie dieser Email ist frei erfunden. Eine Übereinstimmung mit aktuellen oder ehemaligen Regeln wäre rein zufällig und ist nicht beabsichtigt. Tim Weippert <[email protected]> http://www.topf-sicret.org/
Quoted 13 lines · Hugh Dickins
Quoted 8 lines · Tim Weippert
It seems the error is still there:
Aug 29 19:42:09 montdsnsu3 kernel: swap_free: Bad swap file entry c800007fffffc02f Aug 29 19:42:09 montdsnsu3 kernel: Kernel BUG at "mm/rmap.c":493 Aug 29 19:42:09 montdsnsu3 kernel: invalid operand: 0000 [3] SMP Aug 29 19:42:09 montdsnsu3 kernel: Pid: 26550, comm: sh Tainted: G BThanks a lot for trying and reporting back. That's bad news: I was willing to bet that the MSR would fix it. Sorry for wasting your time.
Quoted 7 lines
Not quite conclusive though: I think (from the "[3]" above) that you've not rebooted since the earlier failures? (Nor did I suggest you should.)
It's conceivable (though not very likely) that here you have the error reported on exit from a long-running "sh", running since before you made the MSR fix (the error I'm thinking of occurs when originally exec'ed, but may pass unnoticed while running).
Yes, this can possible, that the sh run before the changes were made. but. The later problem suggest me that this not entirely fix the problem.
Quoted 8 lines
Quoted 4 lines
Bongani Hlope suggest me to try this:
echo 0 >
/proc/sys/kernel/randomize_va_spaceand look for
http://bugzilla.kernel.org/show_bug.cgi?id=4851Please do try that. And if no luck with that, next time it's convenient for you to reboot, please write the MSR as early as you can to see if that makes any difference (probably not, but there's a chance).
Well, now i haven't any entries for the last 5 hours:
Aug 30 08:52:05 montdsnsu3 kernel: ping[6241] general protection rip:2aaaaaaaed43 rsp:7fffff9bff00 error:0 Aug 30 08:54:04 montdsnsu3 kernel: grep[7422] general protection rip:2aaaaaaaed43 rsp:7fffffdbf9b0 error:0 Aug 30 09:14:01 montdsnsu3 kernel: ping[22938] general protection
rip:2aaaaaaaed43 rsp:7fffff9c04b0 error:0
Maybe the randomize_va_space will fix or suppress it ... i will wait until tomorrow and then i think i will reboot the machine and do both, the MSR write and the randomize_va_space settings.
Thanks for your help,
tim
Signature · 6 lines
Interpunktion und Orthographie dieser Email ist frei erfunden. Eine Übereinstimmung mit aktuellen oder ehemaligen Regeln wäre rein zufällig und ist nicht beabsichtigt. Tim Weippert <[email protected]> http://www.topf-sicret.org/
Hello all together,
for now i can confirm, that the problem disappears!
I have done the following:
First try the msr fix, this doesn't solve the problem entirely, but there were no kernel panics.
With the randomize_va_space setting the general protection disappeared too ...
I'm now happy and will go for holiday ( 4 weeks *g*). After that i will look if the machine gets some problems back, but i don't think ...
I thank you all for your help!
Quoted 19 lines · Tim Weippert
Quoted 4 lines · Hugh Dickins
It's conceivable (though not very likely) that here you have the error reported on exit from a long-running "sh", running since before you made the MSR fix (the error I'm thinking of occurs when originally exec'ed, but may pass unnoticed while running).
Yes, this can possible, that the sh run before the changes were made. but. The later problem suggest me that this not entirely fix the problem.
Quoted 8 lines
Quoted 4 lines
Bongani Hlope suggest me to try this:
echo 0 >
/proc/sys/kernel/randomize_va_spaceand look for
http://bugzilla.kernel.org/show_bug.cgi?id=4851Please do try that. And if no luck with that, next time it's convenient for you to reboot, please write the MSR as early as you can to see if that makes any difference (probably not, but there's a chance).
Signature · 4 lines
Every time I think I know where it's at, they move it. Tim Weippert <[email protected]> http://www.topf-sicret.org/