kernel BUG at mm/rmap.c:483!
Hi All,
I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:
Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
No further debug lines are given to diagnose the source of the problem.
I have been using kernel 2.4 for few years now without any problem. This is the first time I see this problem with kernel 2.6. I'm not sure if this is related to the kernel itself, the new hardware, or some other installed software. I'm thinking about downgrading to kernel 2.4. Do you think this will resolve this issue? Any suggestion on what else I can do to mitigate this problem?
Thanks.
-ammar
Quoted 7 lines · Ammar T. Al-Sayegh
Hi All,
I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine
/var/log/messages, I find the following line just before the crash:
Quoted 5 lines
Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!No further debug lines are given to diagnose the
source of the
no oops at all?
which modules are you using?
Quoted 8 lines · Ammar T. Al-Sayegh
Quoted 7 lines
Hi All,
I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine
/var/log/messages, I find the following line just before the crash:
Quoted 6 lines
Quoted 5 lines
Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!No further debug lines are given to diagnose the
source of theno oops at all?
No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?
Is there a way to at least make the server reboot itself next time the kernel is alerted to this problem before crashing? When the server is rebooted, it works fine for few more days before encountering this problem and crashing again.
Quoted 1 line
which modules are you using?
[root ~]# lsmod Module Size Used by
ip_conntrack_ftp 76145 0 md5 8001 1 ipv6 236769 78 autofs4 21829 0 i2c_dev 13249 0 i2c_core 24513 1 i2c_dev sunrpc 135077 1 ipt_REJECT 10561 2 ipt_state 5825 79 ip_conntrack 45317 2 ip_conntrack_ftp,ipt_state iptable_filter 7489 1 ip_tables 20929 3 ipt_REJECT,ipt_state,iptable_filter microcode 11489 0 dm_mod 57925 0 video 19653 0 button 10577 0 battery 13253 0 ac 8773 0 uhci_hcd 33497 0 ehci_hcd 33737 0 e1000 84629 0 floppy 56913 0 ext3 117961 6 jbd 57177 1 ext3 3w_xxxx 30561 0 ata_piix 12485 7 libata 44101 1 ata_piix sd_mod 20545 9 scsi_mod 116033 3 3w_xxxx,libata,sd_mod
Any clue?
-ammar
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 10 lines
Hi All,
I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine
/var/log/messages, I find the following line just before the crash:Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
No further debug lines are given to diagnose the
source of theno oops at all?
No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?
not really; it was supposed to do that already
Quoted 2 lines
i2c_dev 13249 0 i2c_core 24513 1 i2c_dev
try for fun to not use i2c for a while
Quoted 1 line
microcode 11489 0
same for microcode... try removing that so that the microcode of your system doesn't get updated at boot
----- Original Message ----- From: "Arjan van de Ven" <[email protected]>
To: "Ammar T. Al-Sayegh" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, February 23, 2005 5:01 PM
Subject: Re: kernel BUG at mm/rmap.c:483!
Quoted 21 lines · Ammar T. Al-Sayegh
Quoted 11 lines
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 10 lines
Hi All,
I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine
/var/log/messages, I find the following line just before the crash:Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
No further debug lines are given to diagnose the
source of theno oops at all?
No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?
not really; it was supposed to do that already
Quoted 7 lines
Quoted 1 line
i2c_dev13249 0i2c_core24513 1i2c_devtry for fun to not use i2c for a while
Quoted 1 line
microcode 11489 0
same for microcode... try removing that so that the microcode of your system doesn't get updated at boot
What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both i2c_dev and i2c_core or just one of them?
Thanks.
-ammar
Quoted 7 lines
really; it was supposed to do that already
Quoted 6 lines
Quoted 3 lines
i2c_dev 13249 0 i2c_core 24513 1 i2c_devtry for fun to not use i2c for a while
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
microcode 11489 0
same for microcode... try removing that so that the microcode of your system doesn't get updated at boot
What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both
i2c_devandi2c_coreor just one of them?
i2c is used to directly talk to motherboard hardware such as temperature sensors. I've seen cases of certain chipset bugs leading to cacheline corruption when stuff talked to the slow i2c bus and did other stuff in parallel.
microcode changes the microcode of the cpu (a part of your cpu is actually written in "software" that can be updated); however updating this behind the back of the bios might not always be a good idea. (but I have no hard proof of any failures due to this)
As for how to disable these.. you could just rename the respective .ko files to .notko or something....
Quoted 19 lines
Quoted 6 lines
really; it was supposed to do that already
Quoted 5 lines
Quoted 2 lines
i2c_dev13249 0i2c_core24513 1i2c_devtry for fun to not use i2c for a while
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
microcode 11489 0
same for microcode... try removing that so that the microcode of your system doesn't get updated at boot
What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both
i2c_devandi2c_coreor just one of them?i2c is used to directly talk to motherboard hardware such as temperature sensors. I've seen cases of certain chipset bugs leading to cacheline corruption when stuff talked to the slow i2c bus and did other stuff in parallel.
Quoted 7 lines
microcode changes the microcode of the cpu (a part of your cpu is actually written in "software" that can be updated); however updating this behind the back of the bios might not always be a good idea. (but I have no hard proof of any failures due to this)
As for how to disable these.. you could just rename the respective .ko files to .notko or something....
Done. Following is my new loaded module list:
[root ~]# lsmod Module Size Used by
ip_conntrack_ftp 76145 0 md5 8001 1 ipv6 236769 38 autofs4 21829 0 sunrpc 135077 1 ipt_REJECT 10561 2 ipt_state 5825 79 ip_conntrack 45317 2 ip_conntrack_ftp,ipt_state iptable_filter 7489 1 ip_tables 20929 3 ipt_REJECT,ipt_state,iptable_filter dm_mod 57925 0 video 19653 0 button 10577 0 battery 13253 0 ac 8773 0 uhci_hcd 33497 0 ehci_hcd 33737 0 e1000 84629 0 floppy 56913 0 ext3 117961 6 jbd 57177 1 ext3 3w_xxxx 30561 0 ata_piix 12485 7 libata 44101 1 ata_piix sd_mod 20545 9 scsi_mod 116033 3 3w_xxxx,libata,sd_mod
Looks better now?
I guess I can no longer monitor the processor temperature and such after preventing i2c from loading, but what what's the penalty of preventing microcode from loading? a performance hit?
I will be testing memory as suggested by Hugh Dickins as well. Hopefully, your trick or Hugh's suggestion will help revealing the source of the problem, if not the kernel itself.
-ammar
Quoted 3 lines
I guess I can no longer monitor the processor temperature and such after preventing i2c from loading,
yup
Quoted 3 lines
but what what's the
penalty of preventing microcode from loading? a performance hit?
not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines
but what what's the
penalty of preventing microcode from loading? a performance hit?not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..
You can see the processor bugs in intel website, i.e.:
ftp://download.intel.com/design/Xeon/specupdt/24967847.pdf
The following sentence (IMHO) meens that bug is corrected in microcode:
"Workaround: It is possible for the BIOS to contain a workaround for this erratum."
ciao
cate
Quoted 10 lines · Giacomo A. Catenazzi
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines
but what what's the
penalty of preventing microcode from loading? a performance hit?not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..
Quoted 6 lines
You can see the processor bugs in intel website, i.e.:
ftp://download.intel.com/design/Xeon/specupdt/24967847.pdfThe following sentence (IMHO) meens that bug is corrected in microcode:
"Workaround: It is possible for the BIOS to contain a workaround for this erratum."
yeah but it doesn't say in which microcode. Eg it's not possible to find out what a specific microcode update changes over the one the bios already put in...
Quoted 3 lines · Ammar T. Al-Sayegh
I recently installed Fedora RC3 on a new server.
The kernel is 2.6.10-1.741_FC3smp.
I can't really speak for Fedora RC3 kernels, perhaps there's some special patch in there that happens to trigger it for you, but certainly there have been occasional other reports of this BUG with vanilla kernel.org kernels.
Quoted 9 lines
The server
crashes every few days. When I examine/var/log/messages, I find the following line just before the crash:Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------ Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!No further debug lines are given to diagnose the
source of the problem.
It's odd that you get no more lines, but it doesn't really matter in this case. Sadly, the debug info accompanying this BUG has done very little to shed light on its causes (and it's on my todo list to change it to something less of a hindrance).
Quoted 6 lines
I have been using kernel 2.4 for few years now without any problem. This is the first time I see this problem with kernel 2.6. I'm not sure if this is related to the kernel itself, the new hardware, or some other installed software. I'm thinking about downgrading to kernel 2.4. Do you think this will resolve this issue?
Downgrading to 2.4 will most certainly stop that particular BUG, since 2.4 has no equivalent. But it won't necessarily fix the underlying issue.
Quoted 2 lines
Any suggestion on what else I can do to mitigate this problem?
The first thing to do is to give memtest86 a good (say overnight) run. Many of the rmap.c BUG reporters have subsequently found memtest86 failures, and we believe those instances are accounted for by bad memory. And if that's so in your case, you don't really want to be running 2.4 on it.
But not all cases could be accounted in that way. If you report back that memtest86 ran cleanly, then I'll have to rework a debug patch against your Fedora RC3 kernel to try to give us more info - though quite possibly you cannot afford such experiments on this server, and will revert to 2.4 for now.
Hugh
----- Original Message ----- From: "Hugh Dickins" <[email protected]>
To: "Ammar T. Al-Sayegh" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, February 23, 2005 4:31 PM
Subject: Re: kernel BUG at mm/rmap.c:483!
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 3 lines
Any suggestion on what else I can do to mitigate this problem?
The first thing to do is to give memtest86 a good (say overnight) run. Many of the
rmap.cBUG reporters have subsequently found memtest86 failures, and we believe those instances are accounted for by bad memory. And if that's so in your case, you don't really want to be running 2.4 on it.
Quoted 5 lines
But not all cases could be accounted in that way. If you report back that memtest86 ran cleanly, then I'll have to rework a debug patch against your Fedora RC3 kernel to try to give us more info - though quite possibly you cannot afford such experiments on this server, and will revert to 2.4 for now.
The problem is that my server is already in production mode. I'm running great portion of my business on it, where there is very little tolerance for downtime. Because the server is located in a remote datacenter, every time it goes down it takes several hours to have someone sent up there to manually reboot it for a hefty emergency fee. So this bug has already cost me a lot of money, and I'm worried that it will cost me a lot of my clients as well if it persists.
Remote hands are rather expensive, so it will cost me $100/hr to have someone runs memtest86 on my server since I can't perform it remotely. I'll do it though since that's your recommendation for the time being. Hope it will not take more than an hour to run the test, and hope it turns out as bad memory modules as you expect because I hate to downgrade after all the time and money I expended on the upgrade.
-ammar
Quoted 7 lines · Ammar T. Al-Sayegh
----- Original Message ----- From: "Hugh Dickins" <[email protected]>
Quoted 2 lines
though quite possibly you cannot afford
such experiments on this server, and will revert to 2.4 for now.The problem is that my server is already in production mode. I'm running great portion of my business on it, where there is very little tolerance for downtime.
I feared as much.
Quoted 6 lines
Because the server is located in a remote datacenter, every time it goes down it takes several hours to have someone sent up there to manually reboot it for a hefty emergency fee. So this bug has already cost me a lot of money, and I'm worried that it will cost me a lot of my clients as well if it persists.
I'm very sorry for that.
Quoted 8 lines
Remote hands are rather expensive, so it will cost me $100/hr to have someone runs memtest86 on my server since I can't perform it remotely. I'll do it though since that's your recommendation for the time being. Hope it will not take more than an hour to run the test, and hope it turns out as bad memory modules as you expect because I hate to downgrade after all the time and money I expended on the upgrade.
One hour will be enough if it does find a problem in that time, worth a shot; but not enough to give confidence in the memory if it does not find one, 12 hours better. I actually wonder whether rmap.c:483 is the best memory tester (serious answer would be, in some cases yes, but not in all).
Do let me know. If I can find time to rejig the debug patch against your kernel, it would itself keep your server running, replacing the BUG_ON by printks and safety. But without knowing what it will report, I can't judge how satisfactory that would be (and it's unlikely to lead us to the final answer in one go).
Hugh
Quoted 8 lines · Hugh Dickins
Quoted 2 lines · Ammar T. Al-Sayegh
I recently installed Fedora RC3 on a new server.
The kernel is 2.6.10-1.741_FC3smp.I can't really speak for Fedora RC3 kernels, perhaps there's some special patch in there that happens to trigger it for you, but certainly there have been occasional other reports of this BUG with vanilla kernel.org kernels.
That kernel is outdated, current is 2.6.10-1.766_FC3. Before reporting any bugs, update everything! And in case of problems with vendor kernels, it is more useful for everybody involved to report to the distribution.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513