Skip to contentSkip to conversation

kernel BUG at mm/rmap.c:483!

DISCUSSION Ammar T. Al-Sayegh 5 participants 14 messages Feb 23, 2005 – Feb 28, 2005

lore
Share & tools
lore
#0 lore raw

Hi All,

I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!

No further debug lines are given to diagnose the source of the problem.

I have been using kernel 2.4 for few years now without any problem. This is the first time I see this problem with kernel 2.6. I'm not sure if this is related to the kernel itself, the new hardware, or some other installed software. I'm thinking about downgrading to kernel 2.4. Do you think this will resolve this issue? Any suggestion on what else I can do to mitigate this problem?

Thanks.

-ammar

#1 lore raw
Quoted 7 lines · Ammar T. Al-Sayegh

Hi All,

I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Quoted 5 lines
Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!

No further debug lines are given to diagnose the
source of the

no oops at all?

which modules are you using?

#2 lore raw
Quoted 8 lines · Ammar T. Al-Sayegh
Quoted 7 lines

Hi All,

I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Quoted 6 lines
Quoted 5 lines
Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!

No further debug lines are given to diagnose the
source of the

no oops at all?

No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?

Is there a way to at least make the server reboot itself next time the kernel is alerted to this problem before crashing? When the server is rebooted, it works fine for few more days before encountering this problem and crashing again.

Quoted 1 line

which modules are you using?

[root ~]# lsmod Module Size Used by

ip_conntrack_ftp       76145  0 md5                     8001  1 ipv6                  236769  78 autofs4                21829  0 i2c_dev                13249  0 i2c_core               24513  1 i2c_dev
sunrpc                135077  1 ipt_REJECT             10561  2 ipt_state               5825  79 ip_conntrack           45317  2 ip_conntrack_ftp,ipt_state
iptable_filter          7489  1 ip_tables              20929  3 ipt_REJECT,ipt_state,iptable_filter
microcode              11489  0 dm_mod                 57925  0 video                  19653  0 button                 10577  0 battery                13253  0 ac                      8773  0 uhci_hcd               33497  0 ehci_hcd               33737  0 e1000                  84629  0 floppy                 56913  0 ext3                  117961  6 jbd                    57177  1 ext3
3w_xxxx                30561  0 ata_piix               12485  7 libata                 44101  1 ata_piix
sd_mod                 20545  9 scsi_mod              116033  3 3w_xxxx,libata,sd_mod

Any clue?

-ammar

#3 lore raw
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 10 lines

Hi All,

I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

No further debug lines are given to diagnose the
source of the

no oops at all?

No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?

not really; it was supposed to do that already

Quoted 2 lines
i2c_dev                13249  0 
i2c_core               24513  1 i2c_dev

try for fun to not use i2c for a while

Quoted 1 line

microcode 11489 0

same for microcode... try removing that so that the microcode of your system doesn't get updated at boot

#4 lore raw

----- Original Message ----- From: "Arjan van de Ven" <[email protected]>
To: "Ammar T. Al-Sayegh" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, February 23, 2005 5:01 PM
Subject: Re: kernel BUG at mm/rmap.c:483!

Quoted 21 lines · Ammar T. Al-Sayegh
Quoted 11 lines
Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 10 lines

Hi All,

I recently installed Fedora RC3 on a new server. The kernel is 2.6.10-1.741_FC3smp. The server crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

No further debug lines are given to diagnose the
source of the

no oops at all?

No. Is there a way to enable the kernel to give more diagnostic debug output next time this error happens?

not really; it was supposed to do that already

Quoted 7 lines
Quoted 1 line

i2c_dev 13249 0 i2c_core 24513 1 i2c_dev

try for fun to not use i2c for a while

Quoted 1 line

microcode 11489 0

same for microcode... try removing that so that the microcode of your system doesn't get updated at boot

What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both i2c_dev and i2c_core or just one of them?

Thanks.

-ammar

#5 lore raw
Quoted 7 lines

really; it was supposed to do that already

Quoted 6 lines
Quoted 3 lines
i2c_dev                13249  0 
i2c_core               24513  1 i2c_dev

try for fun to not use i2c for a while

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

microcode 11489 0

same for microcode... try removing that so that the microcode of your system doesn't get updated at boot

What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both i2c_dev and i2c_core or just one of them?

i2c is used to directly talk to motherboard hardware such as temperature sensors. I've seen cases of certain chipset bugs leading to cacheline corruption when stuff talked to the slow i2c bus and did other stuff in parallel.

microcode changes the microcode of the cpu (a part of your cpu is actually written in "software" that can be updated); however updating this behind the back of the bios might not always be a good idea. (but I have no hard proof of any failures due to this)

As for how to disable these.. you could just rename the respective .ko files to .notko or something....

#6 lore raw
Quoted 19 lines
Quoted 6 lines

really; it was supposed to do that already

Quoted 5 lines
Quoted 2 lines

i2c_dev 13249 0 i2c_core 24513 1 i2c_dev

try for fun to not use i2c for a while

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

microcode 11489 0

same for microcode... try removing that so that the microcode of your system doesn't get updated at boot

What do these two modules do in particular? and how can I disable them so that they don't get reloaded during boot time? do I need to disable both i2c_dev and i2c_core or just one of them?

i2c is used to directly talk to motherboard hardware such as temperature sensors. I've seen cases of certain chipset bugs leading to cacheline corruption when stuff talked to the slow i2c bus and did other stuff in parallel.

Quoted 7 lines

microcode changes the microcode of the cpu (a part of your cpu is actually written in "software" that can be updated); however updating this behind the back of the bios might not always be a good idea. (but I have no hard proof of any failures due to this)

As for how to disable these.. you could just rename the respective .ko files to .notko or something....

Done. Following is my new loaded module list:

[root ~]# lsmod Module Size Used by

ip_conntrack_ftp       76145  0 md5                     8001  1 ipv6                  236769  38 autofs4                21829  0 sunrpc                135077  1 ipt_REJECT             10561  2 ipt_state               5825  79 ip_conntrack           45317  2 ip_conntrack_ftp,ipt_state
iptable_filter          7489  1 ip_tables              20929  3 ipt_REJECT,ipt_state,iptable_filter
dm_mod                 57925  0 video                  19653  0 button                 10577  0 battery                13253  0 ac                      8773  0 uhci_hcd               33497  0 ehci_hcd               33737  0 e1000                  84629  0 floppy                 56913  0 ext3                  117961  6 jbd                    57177  1 ext3
3w_xxxx                30561  0 ata_piix               12485  7 libata                 44101  1 ata_piix
sd_mod                 20545  9 scsi_mod              116033  3 3w_xxxx,libata,sd_mod

Looks better now?

I guess I can no longer monitor the processor temperature and such after preventing i2c from loading, but what what's the penalty of preventing microcode from loading? a performance hit?

I will be testing memory as suggested by Hugh Dickins as well. Hopefully, your trick or Hugh's suggestion will help revealing the source of the problem, if not the kernel itself.

-ammar

#7 lore raw
Quoted 3 lines

I guess I can no longer monitor the processor temperature and such after preventing i2c from loading,

yup

Quoted 3 lines

but what what's the
penalty of preventing microcode from loading? a performance hit?

not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..

#8 lore raw
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines

but what what's the
penalty of preventing microcode from loading? a performance hit?

not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..

You can see the processor bugs in intel website, i.e.:
ftp://download.intel.com/design/Xeon/specupdt/24967847.pdf

The following sentence (IMHO) meens that bug is corrected in microcode:
"Workaround: It is possible for the BIOS to contain a workaround for this erratum."

ciao
cate

#9 lore raw
Quoted 10 lines · Giacomo A. Catenazzi
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines

but what what's the
penalty of preventing microcode from loading? a performance hit?

not even that; in theory a few cpu bugs may have been fixed. Nobody really knows since there's no changelog for the microcode..

Quoted 6 lines

You can see the processor bugs in intel website, i.e.:
ftp://download.intel.com/design/Xeon/specupdt/24967847.pdf

The following sentence (IMHO) meens that bug is corrected in microcode:
"Workaround: It is possible for the BIOS to contain a workaround for this erratum."

yeah but it doesn't say in which microcode. Eg it's not possible to find out what a specific microcode update changes over the one the bios already put in...

#10 lore raw
In reply to Ammar T. Al-Sayegh
Quoted 3 lines · Ammar T. Al-Sayegh

I recently installed Fedora RC3 on a new server.
The kernel is 2.6.10-1.741_FC3smp.

I can't really speak for Fedora RC3 kernels, perhaps there's some special patch in there that happens to trigger it for you, but certainly there have been occasional other reports of this BUG with vanilla kernel.org kernels.

Quoted 9 lines

The server
crashes every few days. When I examine /var/log/messages, I find the following line just before the crash:

Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!

No further debug lines are given to diagnose the
source of the problem.

It's odd that you get no more lines, but it doesn't really matter in this case. Sadly, the debug info accompanying this BUG has done very little to shed light on its causes (and it's on my todo list to change it to something less of a hindrance).

Quoted 6 lines

I have been using kernel 2.4 for few years now without any problem. This is the first time I see this problem with kernel 2.6. I'm not sure if this is related to the kernel itself, the new hardware, or some other installed software. I'm thinking about downgrading to kernel 2.4. Do you think this will resolve this issue?

Downgrading to 2.4 will most certainly stop that particular BUG, since 2.4 has no equivalent. But it won't necessarily fix the underlying issue.

Quoted 2 lines

Any suggestion on what else I can do to mitigate this problem?

The first thing to do is to give memtest86 a good (say overnight) run. Many of the rmap.c BUG reporters have subsequently found memtest86 failures, and we believe those instances are accounted for by bad memory. And if that's so in your case, you don't really want to be running 2.4 on it.

But not all cases could be accounted in that way. If you report back that memtest86 ran cleanly, then I'll have to rework a debug patch against your Fedora RC3 kernel to try to give us more info - though quite possibly you cannot afford such experiments on this server, and will revert to 2.4 for now.

Hugh

#11 lore raw

----- Original Message ----- From: "Hugh Dickins" <[email protected]>
To: "Ammar T. Al-Sayegh" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, February 23, 2005 4:31 PM
Subject: Re: kernel BUG at mm/rmap.c:483!

Quoted 11 lines · Ammar T. Al-Sayegh
Quoted 3 lines

Any suggestion on what else I can do to mitigate this problem?

The first thing to do is to give memtest86 a good (say overnight) run. Many of the rmap.c BUG reporters have subsequently found memtest86 failures, and we believe those instances are accounted for by bad memory. And if that's so in your case, you don't really want to be running 2.4 on it.

Quoted 5 lines

But not all cases could be accounted in that way. If you report back that memtest86 ran cleanly, then I'll have to rework a debug patch against your Fedora RC3 kernel to try to give us more info - though quite possibly you cannot afford such experiments on this server, and will revert to 2.4 for now.

The problem is that my server is already in production mode. I'm running great portion of my business on it, where there is very little tolerance for downtime. Because the server is located in a remote datacenter, every time it goes down it takes several hours to have someone sent up there to manually reboot it for a hefty emergency fee. So this bug has already cost me a lot of money, and I'm worried that it will cost me a lot of my clients as well if it persists.

Remote hands are rather expensive, so it will cost me $100/hr to have someone runs memtest86 on my server since I can't perform it remotely. I'll do it though since that's your recommendation for the time being. Hope it will not take more than an hour to run the test, and hope it turns out as bad memory modules as you expect because I hate to downgrade after all the time and money I expended on the upgrade.

-ammar

#12 lore raw
Quoted 7 lines · Ammar T. Al-Sayegh

----- Original Message ----- From: "Hugh Dickins" <[email protected]>

Quoted 2 lines

though quite possibly you cannot afford
such experiments on this server, and will revert to 2.4 for now.

The problem is that my server is already in production mode. I'm running great portion of my business on it, where there is very little tolerance for downtime.

I feared as much.

Quoted 6 lines

Because the server is located in a remote datacenter, every time it goes down it takes several hours to have someone sent up there to manually reboot it for a hefty emergency fee. So this bug has already cost me a lot of money, and I'm worried that it will cost me a lot of my clients as well if it persists.

I'm very sorry for that.

Quoted 8 lines

Remote hands are rather expensive, so it will cost me $100/hr to have someone runs memtest86 on my server since I can't perform it remotely. I'll do it though since that's your recommendation for the time being. Hope it will not take more than an hour to run the test, and hope it turns out as bad memory modules as you expect because I hate to downgrade after all the time and money I expended on the upgrade.

One hour will be enough if it does find a problem in that time, worth a shot; but not enough to give confidence in the memory if it does not find one, 12 hours better. I actually wonder whether rmap.c:483 is the best memory tester (serious answer would be, in some cases yes, but not in all).

Do let me know. If I can find time to rejig the debug patch against your kernel, it would itself keep your server running, replacing the BUG_ON by printks and safety. But without knowing what it will report, I can't judge how satisfactory that would be (and it's unlikely to lead us to the final answer in one go).

Hugh

#13 lore raw
In reply to Hugh Dickins
Quoted 8 lines · Hugh Dickins
Quoted 2 lines · Ammar T. Al-Sayegh

I recently installed Fedora RC3 on a new server.
The kernel is 2.6.10-1.741_FC3smp.

I can't really speak for Fedora RC3 kernels, perhaps there's some special patch in there that happens to trigger it for you, but certainly there have been occasional other reports of this BUG with vanilla kernel.org kernels.

That kernel is outdated, current is 2.6.10-1.766_FC3. Before reporting any bugs, update everything! And in case of problems with vendor kernels, it is more useful for everybody involved to report to the distribution.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513