x86-64 bad pmds in 2.6.11.6
Subsystems (4)
[apologies to Andi for getting this twice, I goofed the l-k address
the first time]
I arrived at the office today to find my workstation had this spew in its dmesg buffer..
mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88). mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003). mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b). mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c). mm/memory.c:97: bad pmd ffff81004b017458(00007ffffffff73d). mm/memory.c:97: bad pmd ffff81004b017468(00007ffffffff73e). mm/memory.c:97: bad pmd ffff81004b017470(00007ffffffff73f). mm/memory.c:97: bad pmd ffff81004b017478(00007ffffffff740). mm/memory.c:97: bad pmd ffff81004b017480(00007ffffffff741). mm/memory.c:97: bad pmd ffff81004b017488(00007ffffffff742). mm/memory.c:97: bad pmd ffff81004b017490(00007ffffffff743). mm/memory.c:97: bad pmd ffff81004b017498(00007ffffffff744). mm/memory.c:97: bad pmd ffff81004b0174a0(00007ffffffff745). mm/memory.c:97: bad pmd ffff81004b0174a8(00007ffffffff746). mm/memory.c:97: bad pmd ffff81004b0174b0(00007ffffffff747). mm/memory.c:97: bad pmd ffff81004b0174b8(00007ffffffff748). mm/memory.c:97: bad pmd ffff81004b0174c0(00007ffffffff749). mm/memory.c:97: bad pmd ffff81004b0174c8(00007ffffffff74a). mm/memory.c:97: bad pmd ffff81004b0174d0(00007ffffffff74b). mm/memory.c:97: bad pmd ffff81004b0174d8(00007ffffffff74c). mm/memory.c:97: bad pmd ffff81004b0174e0(00007ffffffff74d). mm/memory.c:97: bad pmd ffff81004b0174e8(00007ffffffff74e). mm/memory.c:97: bad pmd ffff81004b0174f0(00007ffffffff74f). mm/memory.c:97: bad pmd ffff81004b0174f8(00007ffffffff750). mm/memory.c:97: bad pmd ffff81004b017500(00007ffffffff751). mm/memory.c:97: bad pmd ffff81004b017508(00007ffffffff752). mm/memory.c:97: bad pmd ffff81004b017510(00007ffffffff753). mm/memory.c:97: bad pmd ffff81004b017518(00007ffffffff754). mm/memory.c:97: bad pmd ffff81004b017520(00007ffffffff755). mm/memory.c:97: bad pmd ffff81004b017528(00007ffffffff756). mm/memory.c:97: bad pmd ffff81004b017530(00007ffffffff757). mm/memory.c:97: bad pmd ffff81004b017538(00007ffffffff758). mm/memory.c:97: bad pmd ffff81004b017540(00007ffffffff759). mm/memory.c:97: bad pmd ffff81004b017548(00007ffffffff75a). mm/memory.c:97: bad pmd ffff81004b017550(00007ffffffff75b). mm/memory.c:97: bad pmd ffff81004b017558(00007ffffffff75c). mm/memory.c:97: bad pmd ffff81004b017560(00007ffffffff75d). mm/memory.c:97: bad pmd ffff81004b017568(00007ffffffff75e). mm/memory.c:97: bad pmd ffff81004b017570(00007ffffffff75f). mm/memory.c:97: bad pmd ffff81004b017578(00007ffffffff760). mm/memory.c:97: bad pmd ffff81004b017580(00007ffffffff761). mm/memory.c:97: bad pmd ffff81004b017588(00007ffffffff762). mm/memory.c:97: bad pmd ffff81004b017590(00007ffffffff763). mm/memory.c:97: bad pmd ffff81004b017598(00007ffffffff764). mm/memory.c:97: bad pmd ffff81004b0175a0(00007ffffffff765). mm/memory.c:97: bad pmd ffff81004b0175a8(00007ffffffff766). mm/memory.c:97: bad pmd ffff81004b0175b0(00007ffffffff767). mm/memory.c:97: bad pmd ffff81004b0175b8(00007ffffffff768). mm/memory.c:97: bad pmd ffff81004b0175c0(00007ffffffff769). mm/memory.c:97: bad pmd ffff81004b0175c8(00007ffffffff76a). mm/memory.c:97: bad pmd ffff81004b0175d0(00007ffffffff76b). mm/memory.c:97: bad pmd ffff81004b0175d8(00007ffffffff76c). mm/memory.c:97: bad pmd ffff81004b0175e0(00007ffffffff76d). mm/memory.c:97: bad pmd ffff81004b0175e8(00007ffffffff76e). mm/memory.c:97: bad pmd ffff81004b0175f0(00007ffffffff76f). mm/memory.c:97: bad pmd ffff81004b0175f8(00007ffffffff770). mm/memory.c:97: bad pmd ffff81004b017600(00007ffffffff771). mm/memory.c:97: bad pmd ffff81004b017608(00007ffffffff772). mm/memory.c:97: bad pmd ffff81004b017610(00007ffffffff773). mm/memory.c:97: bad pmd ffff81004b017618(00007ffffffff774). mm/memory.c:97: bad pmd ffff81004b017628(0000000000000010). mm/memory.c:97: bad pmd ffff81004b017630(00000000078bfbff). mm/memory.c:97: bad pmd ffff81004b017638(0000000000000006). mm/memory.c:97: bad pmd ffff81004b017640(0000000000001000). mm/memory.c:97: bad pmd ffff81004b017648(0000000000000011). mm/memory.c:97: bad pmd ffff81004b017650(0000000000000064). mm/memory.c:97: bad pmd ffff81004b017658(0000000000000003). mm/memory.c:97: bad pmd ffff81004b017660(0000000000400040). mm/memory.c:97: bad pmd ffff81004b017668(0000000000000004). mm/memory.c:97: bad pmd ffff81004b017670(0000000000000038). mm/memory.c:97: bad pmd ffff81004b017678(0000000000000005). mm/memory.c:97: bad pmd ffff81004b017680(0000000000000008). mm/memory.c:97: bad pmd ffff81004b017688(0000000000000007). mm/memory.c:97: bad pmd ffff81004b017698(0000000000000008). mm/memory.c:97: bad pmd ffff81004b0176a8(0000000000000009). mm/memory.c:97: bad pmd ffff81004b0176b0(0000000000403840). mm/memory.c:97: bad pmd ffff81004b0176b8(000000000000000b). mm/memory.c:97: bad pmd ffff81004b0176c0(00000000000001f4). mm/memory.c:97: bad pmd ffff81004b0176c8(000000000000000c). mm/memory.c:97: bad pmd ffff81004b0176d0(00000000000001f4). mm/memory.c:97: bad pmd ffff81004b0176d8(000000000000000d). mm/memory.c:97: bad pmd ffff81004b0176e0(00000000000001f4). mm/memory.c:97: bad pmd ffff81004b0176e8(000000000000000e). mm/memory.c:97: bad pmd ffff81004b0176f0(00000000000001f4). mm/memory.c:97: bad pmd ffff81004b0176f8(0000000000000017). mm/memory.c:97: bad pmd ffff81004b017708(000000000000000f). mm/memory.c:97: bad pmd ffff81004b017710(00007ffffffff734). mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000). mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).
I've not done a memtest86 run on this (yet), but I'll be very surprised if this is bad RAM, especially considering other folks also seem to have hit the same thing when they moved to 2.6.11. (My workstation ran 2.6.9/2.6.10 without incident previously).
http://lkml.org/lkml/2005/3/11/42 for example lists a similar dump (though obviously differing addresses). Googling around reveals a bunch of other similar dumps.
Dave
Quoted 6 lines · Dave Jones
[apologies to Andi for getting this twice, I goofed the l-k address
the first time]I arrived at the office today to find my workstation had this spew in its dmesg buffer..
Looks like random memory corruption to me.
Can you enable slab debugging etc.?
Quoted 97 lines
mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88).mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003).mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b).mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c).mm/memory.c:97: bad pmd ffff81004b017458(00007ffffffff73d).mm/memory.c:97: bad pmd ffff81004b017468(00007ffffffff73e).mm/memory.c:97: bad pmd ffff81004b017470(00007ffffffff73f).mm/memory.c:97: bad pmd ffff81004b017478(00007ffffffff740).mm/memory.c:97: bad pmd ffff81004b017480(00007ffffffff741).mm/memory.c:97: bad pmd ffff81004b017488(00007ffffffff742).mm/memory.c:97: bad pmd ffff81004b017490(00007ffffffff743).mm/memory.c:97: bad pmd ffff81004b017498(00007ffffffff744).mm/memory.c:97: bad pmd ffff81004b0174a0(00007ffffffff745).mm/memory.c:97: bad pmd ffff81004b0174a8(00007ffffffff746).mm/memory.c:97: bad pmd ffff81004b0174b0(00007ffffffff747).mm/memory.c:97: bad pmd ffff81004b0174b8(00007ffffffff748).mm/memory.c:97: bad pmd ffff81004b0174c0(00007ffffffff749).mm/memory.c:97: bad pmd ffff81004b0174c8(00007ffffffff74a).mm/memory.c:97: bad pmd ffff81004b0174d0(00007ffffffff74b).mm/memory.c:97: bad pmd ffff81004b0174d8(00007ffffffff74c).mm/memory.c:97: bad pmd ffff81004b0174e0(00007ffffffff74d).mm/memory.c:97: bad pmd ffff81004b0174e8(00007ffffffff74e).mm/memory.c:97: bad pmd ffff81004b0174f0(00007ffffffff74f).mm/memory.c:97: bad pmd ffff81004b0174f8(00007ffffffff750).mm/memory.c:97: bad pmd ffff81004b017500(00007ffffffff751).mm/memory.c:97: bad pmd ffff81004b017508(00007ffffffff752).mm/memory.c:97: bad pmd ffff81004b017510(00007ffffffff753).mm/memory.c:97: bad pmd ffff81004b017518(00007ffffffff754).mm/memory.c:97: bad pmd ffff81004b017520(00007ffffffff755).mm/memory.c:97: bad pmd ffff81004b017528(00007ffffffff756).mm/memory.c:97: bad pmd ffff81004b017530(00007ffffffff757).mm/memory.c:97: bad pmd ffff81004b017538(00007ffffffff758).mm/memory.c:97: bad pmd ffff81004b017540(00007ffffffff759).mm/memory.c:97: bad pmd ffff81004b017548(00007ffffffff75a).mm/memory.c:97: bad pmd ffff81004b017550(00007ffffffff75b).mm/memory.c:97: bad pmd ffff81004b017558(00007ffffffff75c).mm/memory.c:97: bad pmd ffff81004b017560(00007ffffffff75d).mm/memory.c:97: bad pmd ffff81004b017568(00007ffffffff75e).mm/memory.c:97: bad pmd ffff81004b017570(00007ffffffff75f).mm/memory.c:97: bad pmd ffff81004b017578(00007ffffffff760).mm/memory.c:97: bad pmd ffff81004b017580(00007ffffffff761).mm/memory.c:97: bad pmd ffff81004b017588(00007ffffffff762).mm/memory.c:97: bad pmd ffff81004b017590(00007ffffffff763).mm/memory.c:97: bad pmd ffff81004b017598(00007ffffffff764).mm/memory.c:97: bad pmd ffff81004b0175a0(00007ffffffff765).mm/memory.c:97: bad pmd ffff81004b0175a8(00007ffffffff766).mm/memory.c:97: bad pmd ffff81004b0175b0(00007ffffffff767).mm/memory.c:97: bad pmd ffff81004b0175b8(00007ffffffff768).mm/memory.c:97: bad pmd ffff81004b0175c0(00007ffffffff769).mm/memory.c:97: bad pmd ffff81004b0175c8(00007ffffffff76a).mm/memory.c:97: bad pmd ffff81004b0175d0(00007ffffffff76b).mm/memory.c:97: bad pmd ffff81004b0175d8(00007ffffffff76c).mm/memory.c:97: bad pmd ffff81004b0175e0(00007ffffffff76d).mm/memory.c:97: bad pmd ffff81004b0175e8(00007ffffffff76e).mm/memory.c:97: bad pmd ffff81004b0175f0(00007ffffffff76f).mm/memory.c:97: bad pmd ffff81004b0175f8(00007ffffffff770).mm/memory.c:97: bad pmd ffff81004b017600(00007ffffffff771).mm/memory.c:97: bad pmd ffff81004b017608(00007ffffffff772).mm/memory.c:97: bad pmd ffff81004b017610(00007ffffffff773).mm/memory.c:97: bad pmd ffff81004b017618(00007ffffffff774).mm/memory.c:97: bad pmd ffff81004b017628(0000000000000010).mm/memory.c:97: bad pmd ffff81004b017630(00000000078bfbff).mm/memory.c:97: bad pmd ffff81004b017638(0000000000000006).mm/memory.c:97: bad pmd ffff81004b017640(0000000000001000).mm/memory.c:97: bad pmd ffff81004b017648(0000000000000011).mm/memory.c:97: bad pmd ffff81004b017650(0000000000000064).mm/memory.c:97: bad pmd ffff81004b017658(0000000000000003).mm/memory.c:97: bad pmd ffff81004b017660(0000000000400040).mm/memory.c:97: bad pmd ffff81004b017668(0000000000000004).mm/memory.c:97: bad pmd ffff81004b017670(0000000000000038).mm/memory.c:97: bad pmd ffff81004b017678(0000000000000005).mm/memory.c:97: bad pmd ffff81004b017680(0000000000000008).mm/memory.c:97: bad pmd ffff81004b017688(0000000000000007).mm/memory.c:97: bad pmd ffff81004b017698(0000000000000008).mm/memory.c:97: bad pmd ffff81004b0176a8(0000000000000009).mm/memory.c:97: bad pmd ffff81004b0176b0(0000000000403840).mm/memory.c:97: bad pmd ffff81004b0176b8(000000000000000b).mm/memory.c:97: bad pmd ffff81004b0176c0(00000000000001f4).mm/memory.c:97: bad pmd ffff81004b0176c8(000000000000000c).mm/memory.c:97: bad pmd ffff81004b0176d0(00000000000001f4).mm/memory.c:97: bad pmd ffff81004b0176d8(000000000000000d).mm/memory.c:97: bad pmd ffff81004b0176e0(00000000000001f4).mm/memory.c:97: bad pmd ffff81004b0176e8(000000000000000e).mm/memory.c:97: bad pmd ffff81004b0176f0(00000000000001f4).mm/memory.c:97: bad pmd ffff81004b0176f8(0000000000000017).mm/memory.c:97: bad pmd ffff81004b017708(000000000000000f).mm/memory.c:97: bad pmd ffff81004b017710(00007ffffffff734).mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000).mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).I've not done a memtest86 run on this (yet), but I'll be very surprised if this is bad RAM, especially considering other folks also seem to have hit the same thing when they moved to 2.6.11. (My workstation ran 2.6.9/2.6.10 without incident previously).
Quoted 3 lines
http://lkml.org/lkml/2005/3/11/42 for example lists a similar dump (though obviously differing addresses). Googling around reveals a bunch of other similar dumps.
Yes I saw them, but I supposed it is some driver going bad. If you want you can collect hardware data and see if there is a common driver.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Dave Jones
[apologies to Andi for getting this twice, I goofed the l-k address
the first time]I arrived at the office today to find my workstation had this spew in its dmesg buffer..
Looks like random memory corruption to me.
Can you enable slab debugging etc.?
SLAB_DEBUG=y. Nothing in the logs.
Quoted 3 lines
Yes I saw them, but I supposed it is some driver going bad. If you want you can collect hardware data and see if there is a common driver.
There's quite a bit in this box
00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:07.5 Multimedia audio controller: Advanced Micro Devices [AMD] AMD-8111 AC97 Audio (rev 03)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
02:07.0 USB Controller: NEC Corporation USB (rev 41)
02:07.1 USB Controller: NEC Corporation USB (rev 41)
02:07.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
02:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01)
02:08.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01)
02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:0a.0 Unknown mass storage controller: Triones Technologies, Inc. HPT366/368/370/370A/372 (rev 03)
03:0b.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
03:0c.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
04:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-8151 System Controller (rev 13)
04:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8151 AGP Bridge (rev 13)
05:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G550 AGP (rev 01)
The SATA & SCSI controllers have no disks attached. Firewire can be ignored (theres no actual connector even for it on the board). The various USB controllers are mostly unused. Only one of them is USB2.0, so that sees occasional usb-storage use. Not noticed anything going bad there though.
Dave
Quoted 51 lines · Dave Jones
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Dave Jones
[apologies to Andi for getting this twice, I goofed the l-k address
the first time]I arrived at the office today to find my workstation had this spew in its dmesg buffer..
Looks like random memory corruption to me.
Can you enable slab debugging etc.?
SLAB_DEBUG=y. Nothing in the logs.
Quoted 3 lines
Yes I saw them, but I supposed it is some driver going bad. If you want you can collect hardware data and see if there is a common driver.
There's quite a bit in this box
00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:07.5 Multimedia audio controller: Advanced Micro Devices [AMD] AMD-8111 AC97 Audio (rev 03)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
02:07.0 USB Controller: NEC Corporation USB (rev 41)
02:07.1 USB Controller: NEC Corporation USB (rev 41)
02:07.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
02:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01)
02:08.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01)
02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:0a.0 Unknown mass storage controller: Triones Technologies, Inc. HPT366/368/370/370A/372 (rev 03)
03:0b.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
03:0c.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
04:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-8151 System Controller (rev 13)
04:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8151 AGP Bridge (rev 13)
05:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G550 AGP (rev 01)
Quoted 6 lines
The SATA & SCSI controllers have no disks attached. Firewire can be ignored (theres no actual connector even for it on the board). The various USB controllers are mostly unused. Only one of them is USB2.0, so that sees occasional usb-storage use. Not noticed anything going bad there though.
Dave
And here is my box (looks like there is no many hardware drivers in common).
rathamahata@lights rathamahata $ /sbin/lspci
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 0000:01:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 02)
0000:03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller 0000:03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller 0000:04:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:04:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:04:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
rathamahata@lights rathamahata $
e1000 is handled by by intel's e1000 driver
usb is not compiled in
rathamahata@lights linux-2.6.11 $ grep CONFIG_USB .config # CONFIG_USB is not set CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y
# CONFIG_USB_GADGET is not set rathamahata@lights linux-2.6.11 $
Signature · 2 lines
Sergey S. Kostyliov <[email protected]> Jabber ID: [email protected]
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Dave Jones
[apologies to Andi for getting this twice, I goofed the l-k address
the first time]
Quoted 12 lines
Quoted 2 lines
I arrived at the office today to find my workstation had this spew in its dmesg buffer..
Looks like random memory corruption to me.
Can you enable slab debugging etc.?
Quoted 5 lines
mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88).mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003).mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b).mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c).
etc..
I realised today that this happens every time X starts up for
the first time. I did some experiments, and found that with 2.6.12rc1
it's gone. Either it got fixed accidentally, or its hidden now by one of the many changes in 4-level patches.
I'll try and narrow this down a little more tomorrow, to see if I can pinpoint the exact -bk snapshot (may be tricky given they were broken for a while), as it'd be good to get this fixed in 2.6.11.x if .12 isn't going to show up any time soon.
Dave
Quoted 9 lines
I realised today that this happens every time X starts up for
the first time. I did some experiments, and found that with 2.6.12rc1
it's gone. Either it got fixed accidentally, or its hidden now by one of the many changes in 4-level patches.I'll try and narrow this down a little more tomorrow, to see if I can pinpoint the exact -bk snapshot (may be tricky given they were broken for a while), as it'd be good to get this fixed in 2.6.11.x if .12 isn't going to show up any time soon.
Can you supply a strace of the /dev/mem, /dev/kmem accesses of your X server? (including the mmaps or read/writes if available)
My X server doesn't seem to cause that.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 9 lines · Dave Jones
I realised today that this happens every time X starts up for
the first time. I did some experiments, and found that with 2.6.12rc1
it's gone. Either it got fixed accidentally, or its hidden now by one of the many changes in 4-level patches.I'll try and narrow this down a little more tomorrow, to see if I can pinpoint the exact -bk snapshot (may be tricky given they were broken for a while), as it'd be good to get this fixed in 2.6.11.x if .12 isn't going to show up any time soon.
Quoted 4 lines
Can you supply a strace of the
/dev/mem,/dev/kmemaccesses of your X server? (including the mmaps or read/writes if available)My X server doesn't seem to cause that.
I can't explain why it should appear fixed in 2.6.12-rc1 (probably other complicating factors at work), but I do believe you've fixed this in 2.6.12-rc2, and the patch which should go into -stable is your load_cr3 patch below, which Linus took from Andrew on 28 March.
I say this because I was intrigued by the resemblance between Sergey's and Dave's corruptions, and spent a while trying to work out where they come from. The giveaway is the little ASCII string they share at the end (seen also in Clem's extract)
mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000). mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).
That says "x86_64", and a grep for that as a string shows ELF_PLATFORM, and a grep for that shows create_elf_tables in fs/binfmt_elf.c. _All_ this pmd corruption (except for the first line, presumably pushing a user address on stack) originates from create_elf_tables (the neatly ascending stack addresses being the argv and envp pointers, incrementing by 1 because only a NUL-string is found for each, the real strings being off elsewhere in the intended new stack page, not in this pmd page).
It looks very much as if the mm being created has for pmd a page which was used for user stack in the outgoing mm; but somehow exec's exit_mmap TLB flushing hasn't taken effect. I only now noticed this patch where you fix just such an issue.
Hugh
From: "Andi Kleen" <[email protected]>
Always reload CR3 completely when a lazy MM thread drops a MM. This avoids keeping stale mappings around in the TLB that could be run into by the CPU by itself (e.g. during prefetches).
---
25-akpm/arch/x86_64/kernel/smp.c | 3 ++- 25-akpm/include/asm-x86_64/mmu_context.h | 10 ++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)
arch/x86_64/kernel/smp.c renamed from arch/x86_64/kernel/smp.c~x86_64-always-reload-cr3-completely-when-a-lazy-mm +2 -1
@@ -25,6 +25,7 @@#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include <asm/mach_apic.h> +#include <asm/mmu_context.h> #include <asm/proto.h> /*
@@ -52,7 +53,7 @@ static inline void leave_mm (unsigned loif (read_pda (mmu_state) ==TLBSTATE_OK )BUG ();clear_bit (cpu, &read_pda (active_mm)->cpu_vm_mask); -__flush_tlb (); +load_cr3 (swapper_pg_dir); }/*
include/asm-x86_64/mmu_context.h renamed from include/asm-x86_64/mmu_context.h~x86_64-always-reload-cr3-completely-when-a-lazy-mm +8 -2
@@ -28,6 +28,11 @@ static inline void enter_lazy_tlb(struct }#endif +static inline void load_cr3 (pgd_t *pgd) +{ + asmvolatile ("movq %0,%%cr3" ::"r" (__pa (pgd)) :"memory" ); +} +static inline void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk) {
@@ -40,7 +45,8 @@ static inline void switch_mm(struct mm_swrite_pda (active_mm, next);#endif set_bit (cpu, &next->cpu_vm_mask); - asmvolatile ("movq %0,%%cr3" ::"r" (__pa (next->pgd)) :"memory" ); +load_cr3 (next->pgd); +if (unlikely (next->context.ldt != prev->context.ldt))load_LDT_nolock (&next->context, cpu); }
@@ -54,7 +60,7 @@ static inline void switch_mm(struct mm_s* tlb flush IPI delivery. We must reload CR3 * to make sure to use no freed page tables. */ - asmvolatile ("movq %0,%%cr3" ::"r" (__pa (next->pgd)) :"memory" ); +load_cr3 (next->pgd);load_LDT_nolock (&next->context, cpu); } }
Quoted 4 lines
It looks very much as if the mm being created has for pmd a page which was used for user stack in the outgoing mm; but somehow exec's
exit_mmapTLB flushing hasn't taken effect. I only now noticed this patch where you fix just such an issue.
Thanks for the analysis. However I doubt the load_cr3 patch can fix it. All it does is to stop the CPU from prefetching mappings (which can cause different problem). But the Linux code who does bad pmd checks never looks at CR3 anyways, it always uses the current->mm. If bad pmd sees a bad page it must be still in the page tables of the MM, not a stable TLB entry.
It must be something else. Somehow we get a freed page into the page table hierarchy. After the initial 4level implementation I did not do many changes there, my suspection would be rather on the recent memory.c changes.
-Andi
Quoted 4 lines · Andi Kleen
Thanks for the analysis. However I doubt the
load_cr3patch can fix it. All it does is to stop the CPU from prefetching mappings (which can cause different problem).
I thought that the leave_mm code (before your patch) flushes the TLB, but restores cr3 to the mm, while removing that cpu from the mm's cpu_vm_mask.
So any speculation, not just prefetching, on that cpu is in danger of bringing address translations according to that mm back into the TLB.
But when the mm is torn down in exit_mmap, there's no longer any record that the TLB on that cpu needs flushing, so stale translations remain.
As a rule, we always flush TLB _after_ invalidating, not just before, for this kind of reason.
My paranoia of speculation may be excessive: I _think_ what I outline above is a real possibility on Intel, but you and others know AMD much better than I (and the reports I've seen are on AMD64, not EM64T).
Quoted 4 lines
But the Linux code who does bad pmd checks
never looks at CR3 anyways, it always uses thecurrent->mm. If bad pmd sees a bad page it must be still in the page tables of the MM, not a stable TLB entry.
Sure, the "mm/memory.c:97: bad pmd" messages are coming from clear_pmd_range, when the corrupted task exits later (but probably not much later, since its user stack is oddly distributed across two different pages: some mentioned SIGSEGVs I think).
The pmd really is bad, but it got to be bad because it had stack data written into it by create_elf_tables, when the TLB mistakenly thought it already knew what physical page 0x00007ffffffff000 was mapped to (prior kernel accesses to that user stack are not by user address).
Hugh
Quoted 9 lines · Hugh Dickins
Quoted 4 lines · Andi Kleen
Thanks for the analysis. However I doubt the
load_cr3patch can fix it. All it does is to stop the CPU from prefetching mappings (which can cause different problem).I thought that the
leave_mmcode (before your patch) flushes the TLB, but restores cr3 to the mm, while removing that cpu from the mm'scpu_vm_mask.
Quoted 8 lines
So any speculation, not just prefetching, on that cpu is in danger of bringing address translations according to that mm back into the TLB.
But when the mm is torn down in
exit_mmap, there's no longer any record that the TLB on that cpu needs flushing, so stale translations remain.As a rule, we always flush TLB _after_ invalidating, not just before, for this kind of reason.
Yes this is all true. In fact I have several bug fixes for problems in this area.
But this all cannot explain corruptions comming from the kernel, you tend to only see problems with the CPU prefetching something.
Note that with the cr3 reload you end up with init_mm, which is not any useful mm. So even if there was a store from the kernel into a stale mapping it would cause -EFAULT now. But that is not happening.
Quoted 4 lines
My paranoia of speculation may be excessive: I _think_ what I outline above is a real possibility on Intel, but you and others know AMD much better than I (and the reports I've seen are on AMD64, not EM64T).
It is not both on Intel and AMD :) These CPUs do a lot of prefetching behind your back, any stale mappings at any time in the TLB eventually cause problems. But other ones than this.
Quoted 9 lines
Sure, the "
mm/memory.c:97: bad pmd" messages are coming fromclear_pmd_range, when the corrupted task exits later (but probably not much later, since its user stack is oddly distributed across two different pages: some mentioned SIGSEGVs I think).The pmd really is bad, but it got to be bad because it had stack data written into it by
create_elf_tables, when the TLB mistakenly thought it already knew what physical page0x00007ffffffff000was mapped to (prior kernel accesses to that user stack are not by user address).
What I meant is that the overwriting must be from Linux code acting in the direct mapping, not due stale TLBs for addresses < __PAGE_OFFSET.
I will take a closer look at the rc1/rc2 patches later this evening and see if I can spot something. Can only report back tomorrow though.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 2 lines
I will take a closer look at the rc1/rc2 patches later this evening and see if I can spot something. Can only report back tomorrow though.
Actually itt started in .11 already - sigh - on rereading the thread. That will make the code audit harder :/
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 5 lines · * Andi Kleen ([email protected]) wrote:
Quoted 2 lines
I will take a closer look at the rc1/rc2 patches later this evening and see if I can spot something. Can only report back tomorrow though.
Actually itt started in .11 already - sigh - on rereading the thread. That will make the code audit harder :/
Yes, I've seen it in .11 and earlier kernels. Happen to have same "x86_64" string on my bad pmd dumps, but can't reproduce it at all. So, for now, I can hold off on adding the reload cr3 patch to -stable unless you think it should be there anyway.
thanks,
-chris
Signature · 1 line
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 11 lines · Chris Wright
Quoted 5 lines · * Andi Kleen ([email protected]) wrote:
Quoted 2 lines
I will take a closer look at the rc1/rc2 patches later this evening and see if I can spot something. Can only report back tomorrow though.
Actually itt started in .11 already - sigh - on rereading the thread. That will make the code audit harder :/
Yes, I've seen it in .11 and earlier kernels. Happen to have same "
x86_64" string on my bad pmd dumps, but can't reproduce it at all. So, for now, I can hold off on adding the reload cr3 patch to -stable unless you think it should be there anyway.
It is a bug fix (actually there is another related patch that fixes a similar bug), but we lived with the problems for years so I guess they can wait for .12.
If there was a fix for the bad pmd problem it might be a candidate for stable, but so far we dont know what causes it yet.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 9 lines · * Andi Kleen ([email protected]) wrote:
Quoted 4 lines · Chris Wright
Yes, I've seen it in .11 and earlier kernels. Happen to have same "
x86_64" string on my bad pmd dumps, but can't reproduce it at all. So, for now, I can hold off on adding the reload cr3 patch to -stable unless you think it should be there anyway.It is a bug fix (actually there is another related patch that fixes a similar bug), but we lived with the problems for years so I guess they can wait for .12.
Sounds good.
Quoted 2 lines
If there was a fix for the bad pmd problem it might be a candidate for stable, but so far we dont know what causes it yet.
If I figure a way to trigger here, I'll report back.
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 7 lines · Chris Wright
Quoted 6 lines · * Andi Kleen ([email protected]) wrote:
Quoted 4 lines · Chris Wright
Yes, I've seen it in .11 and earlier kernels. Happen to have same "
x86_64" string on my bad pmd dumps, but can't reproduce it at all. So, for now, I can hold off on adding the reload cr3 patch to -stable unless you think it should be there anyway.
Quoted 5 lines
Quoted 3 lines
It is a bug fix (actually there is another related patch that fixes a similar bug), but we lived with the problems for years so I guess they can wait for .12.
Sounds good.
I must confess, with all due respect to Andi, that I don't understand his dismissal of the possibility that load_cr3 in leave_mm might be the fix (to create_elf_tables writing user stack data into the pmd).
My belief is that leaving any opening for unpredictable speculations to pull stale translations into the TLB, is a recipe for strange trouble down the line when those translations may get used in actuality.
I'd been hoping Andi would come to see it my way overnight! since I'm clearly not up to arguing the case persuasively.
But I certainly don't expect Chris to add an unjustified patch to -stable.
Quoted 4 lines
Quoted 2 lines
If there was a fix for the bad pmd problem it might be a candidate for stable, but so far we dont know what causes it yet.
If I figure a way to trigger here, I'll report back.
Dave, earlier on you were quite able to reproduce the problem on 2.6.11, finding it happened the first time you ran X. Do you have any time to reverify that, then try to reproduce with the load_cr3 in leave_mm patch?
But please don't waste your time on this unless you think it's plausible.
Thanks,
Hugh
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 9 lines · Hugh Dickins
Quoted 3 lines
Quoted 2 lines
If there was a fix for the bad pmd problem it might be a candidate for stable, but so far we dont know what causes it yet.
If I figure a way to trigger here, I'll report back.
Dave, earlier on you were quite able to reproduce the problem on 2.6.11, finding it happened the first time you ran X. Do you have any time to reverify that, then try to reproduce with the
load_cr3inleave_mmpatch?But please don't waste your time on this unless you think it's plausible.
I used to be able to reproduce it 100% by doing this on an vanilla upstream kernel. Then it changed behaviour so I only saw it happening on the Fedora kernel. For the latest Fedora update kernel I backported this change..
- x86_64: Only free PMDs and PUDs after other CPUs have been flushed as a 'try it and see'. At first I thought it killed the bug, but a day or so later, it started doing it again.
In the Fedora kernel we have a patch which restricts /dev/mem reading, so I got suspicious about this interacting with any of the changes that had happened to drivers/char/mem.c
Out of curiousity, I backported the 3-4 patches from .12rc to the Fedora .11 kernel, and haven't seen the problem since.
The bizarre thing is I can't explain why any of those patches would make such a difference. Given the bug seems to be coming and going for me, its possible its just masked the problem.
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Hugh DickinsQuoted 12 lines · Hugh Dickins
Quoted 11 lines · Chris Wright
Quoted 9 lines · * Andi Kleen ([email protected]) wrote:
Quoted 4 lines · Chris Wright
Yes, I've seen it in .11 and earlier kernels. Happen to have same "
x86_64" string on my bad pmd dumps, but can't reproduce it at all. So, for now, I can hold off on adding the reload cr3 patch to -stable unless you think it should be there anyway.It is a bug fix (actually there is another related patch that fixes a similar bug), but we lived with the problems for years so I guess they can wait for .12.
Quoted 5 lines
Quoted 1 line
Sounds good.
I must confess, with all due respect to Andi, that I don't understand his dismissal of the possibility that
load_cr3inleave_mmmight be the fix (tocreate_elf_tableswriting user stack data into the pmd).
Sorry for the late answer.
Ok, lets try again. The hole fixed by this patch only covers the case of an kernel thread with lazy mm doing some memory access (or more likely the CPU doing a prefetch there). But ELF loading never happens in lazy mm kernel threads.AFAIK in a "real" process the TLB is always fully consistent.
Does that explanation satisfy you?
I agree that my earlier one was a bit dubious because I argued about the direct mapping, but the argv setup actually uses user addresses. But I still think it must be something else.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 7 lines · Andi Kleen
Quoted 4 lines · Hugh Dickins
I must confess, with all due respect to Andi, that I don't understand his dismissal of the possibility that
load_cr3inleave_mmmight be the fix (tocreate_elf_tableswriting user stack data into the pmd).Sorry for the late answer.
Not at all. I didn't expect you to persist in trying to persuade me, thank you for doing so, and I apologize for taking your time on this.
Quoted 7 lines
Ok, lets try again. The hole fixed by this patch only covers the case of an kernel thread with lazy mm doing some memory access (or more likely the CPU doing a prefetch there). But ELF loading never happens in lazy mm kernel threads.AFAIK in a "real" process the TLB is always fully consistent.
Does that explanation satisfy you?
It does. Well, I needed to restudy exec_mmap and switch_mm in detail, and having done so, I agree that the only way you can get through exec_mmap's activate_mm without fully flushing the cpu's TLB, is if the active_mm matches the newly allocated mm (itself impossible since there's a reference on the active_mm), and the cpu bit is still set in cpu_vm_mask - precisely not the case if we went through leave_mm. Yet I was claiming your leave_mm fix could flush TLB for exec_mmap where it wasn't already done.
Sorry for letting the neatness of my pmd/stack story blind me to its impossibility, and for wasting your time.
Hugh
Debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Dave JonesCan people who can reproduce the x86-64 2.6.11 pmd bad problem please apply the following patch and see (a) if it can be still reprocuded with it and send the output generated. Also a strace of the program that showed it (pid and name of it should be dumped) would be useful if not too big.
After staring some time at the code I cant find the problem, but I somehow suspect it has to do with early page table frees. That is why they were disabled. This should not cause any memory leaks, the page tables will be always freed at process exit, so it is safe to apply even for production machines.
Thanks,
-Andi
mm/memory.c renamed from mm/memory.c-o +1 -0
@@ -94,6 +94,7 @@if (pmd_none (*pmd))return ;if (unlikely (pmd_bad (*pmd))) { +printk ("%s:%d: " , current->comm, current->pid);pmd_ERROR (*pmd);pmd_clear (pmd);return ;
mm/mmap.c renamed from mm/mmap.c-o +2 -0
@@ -1645,11 +1645,13 @@return ;if (first <FIRST_USER_PGD_NR *PGDIR_SIZE ) first =FIRST_USER_PGD_NR *PGDIR_SIZE ; +#if 0 /* No point trying to free anything if we're in the same pte page */ if ((first &PMD_MASK ) < (last &PMD_MASK )) {clear_page_range (tlb, first, last);flush_tlb_pgtables (mm, first, last); } +#endif }/* Normal function to fix up a mapping
New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Dave JonesCould someone who reproduces this problem apply the following patch and see if the WARN_ON triggers?
mm/memory.c renamed from mm/memory.c-o +2 -0
@@ -94,6 +94,7 @@if (pmd_none (*pmd))return ;if (unlikely (pmd_bad (*pmd))) { +printk ("%s:%d: " , current->comm, current->pid);pmd_ERROR (*pmd);pmd_clear (pmd);return ;
@@ -113,6 +114,7 @@unsigned long addr = start, next;pmd_t *pmd, *__pmd; +WARN_ON (start == end);if (pud_none (*pud))return ;if (unlikely (pud_bad (*pud))) {
Re: New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 17 lines · Andi Kleen
Could someone who reproduces this problem apply the following patch and see if the
WARN_ONtriggers?diff -u linux-2.6.11/mm/memory.c-o
linux-2.6.11/mm/memory.cmm/memory.c renamed from mm/memory.c-o +1
-0@@ -94,6 +94,7 @@if (pmd_none (*pmd))return ;if (unlikely (pmd_bad (*pmd))) { +printk ("%s:%d: " , current->comm, current->pid);pmd_ERROR (*pmd);pmd_clear (pmd);return ;
mm/memory.c line 120
Quoted 7 lines
mm/memory.c renamed from mm/memory.c-o +1
-0@@ -113,6 +114,7 @@unsigned long addr = start, next;pmd_t *pmd, *__pmd; +WARN_ON (start == end);if (pud_none (*pud))return ;if (unlikely (pud_bad (*pud))) {
I'm up to my eyeballs in other stuff right now, so probably won't get a chance to test this personally. I'll add it to the Fedora testing rpm however, as 1-2 users are also hitting it.
I'll let you know if I hear anything back.
Dave
Re: New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 5 lines · Dave Jones
Quoted 4 lines · Andi Kleen
Could someone who reproduces this problem apply the following patch and see if the
WARN_ONtriggers?
*snip*
Quoted 7 lines
I'm up to my eyeballs in other stuff right now, so probably won't get a chance to test this personally. I'll add it to the Fedora testing rpm however, as 1-2 users are also hitting it.
I'll let you know if I hear anything back.
Dave
I'm seeing this problem on 2.6.11-1.14_FC3smp, a part of dmesg is supplied below. (these messages have flooded the buffer). Kernel was updated from kernel-smp-2.6.9-1.667 yesterday, and no similar messages was in dmesg at that time after 8 days uptime.
Fedora Core 3 w/latest updates atm
Running Mysql, exim, http + perl, clamav, spamassasin, courier.
Tyan motherboard, dual AMD 246, 4x512MB ram (one on each bus), two 3ware 9xx, dual broadcom nic. (lspci output below)
I have not tried the testing kernel you suggested, but will see what time will allow today. We already have a scheduled reboot on it so maybe I'll just sneak in a kernel update again.
More info available on request.
-HK
# uptime 12:57:50 up 4:03, 1 user, load average: 1.85, 1.87, 1.74
# lspci 00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID 02:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 03:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 03:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 10)
# lsmod Module Size Used by md5 5953 1 ipv6 297665 52 autofs4 24521 0 sunrpc 169017 1 xfs 563953 1 exportfs 8385 1 xfs dm_mod 69761 0 video 20169 0 button 9185 0 battery 12233 0 ac 6857 0 i2c_amd8111 8129 0 i2c_core 28353 1 i2c_amd8111 hw_random 7393 0 e100 42305 0 mii 7105 1 e100 tg3 91717 0 reiserfs 276665 3 3w_9xxx 38725 6 sd_mod 20929 8 scsi_mod 155665 2 3w_9xxx,sd_mod
Part of dmesg:mm/memory.c:97: bad pmd ffff810059861a80(000000000000000b). mm/memory.c:97: bad pmd ffff810059861a90(000000000000000c). mm/memory.c:97: bad pmd ffff810059861aa0(000000000000000d). mm/memory.c:97: bad pmd ffff810059861ab0(000000000000000e). mm/memory.c:97: bad pmd ffff810059861ac0(0000000000000017). mm/memory.c:97: bad pmd ffff810059861ad0(000000000000000f). mm/memory.c:97: bad pmd ffff810059861ad8(00007ffffffffaf8). mm/memory.c:97: bad pmd ffff810059861af8(000034365f363878). mm/memory.c:97: bad pmd ffff810074cb7878(0000003926400a88). mm/memory.c:97: bad pmd ffff810074cb7880(0000000000000004). mm/memory.c:97: bad pmd ffff810074cb7888(00007ffffffffb00). mm/memory.c:97: bad pmd ffff810074cb7890(00007ffffffffb01). mm/memory.c:97: bad pmd ffff810074cb7898(00007ffffffffb02). mm/memory.c:97: bad pmd ffff810074cb78a0(00007ffffffffb03). mm/memory.c:97: bad pmd ffff810074cb78b0(00007ffffffffb04). mm/memory.c:97: bad pmd ffff810074cb78b8(00007ffffffffb05). mm/memory.c:97: bad pmd ffff810074cb78c0(00007ffffffffb06). mm/memory.c:97: bad pmd ffff810074cb78c8(00007ffffffffb07). mm/memory.c:97: bad pmd ffff810074cb78d0(00007ffffffffb08). mm/memory.c:97: bad pmd ffff810074cb78d8(00007ffffffffb09). mm/memory.c:97: bad pmd ffff810074cb78e0(00007ffffffffb0a). mm/memory.c:97: bad pmd ffff810074cb78e8(00007ffffffffb0b). mm/memory.c:97: bad pmd ffff810074cb78f0(00007ffffffffb0c). mm/memory.c:97: bad pmd ffff810074cb78f8(00007ffffffffb0d). mm/memory.c:97: bad pmd ffff810074cb7900(00007ffffffffb0e). mm/memory.c:97: bad pmd ffff810074cb7908(00007ffffffffb0f). mm/memory.c:97: bad pmd ffff810074cb7910(00007ffffffffb10). mm/memory.c:97: bad pmd ffff810074cb7918(00007ffffffffb11). mm/memory.c:97: bad pmd ffff810074cb7920(00007ffffffffb12). mm/memory.c:97: bad pmd ffff810074cb7928(00007ffffffffb13). mm/memory.c:97: bad pmd ffff810074cb7930(00007ffffffffb14). mm/memory.c:97: bad pmd ffff810074cb7938(00007ffffffffb15). mm/memory.c:97: bad pmd ffff810074cb7940(00007ffffffffb16). mm/memory.c:97: bad pmd ffff810074cb7948(00007ffffffffb17). mm/memory.c:97: bad pmd ffff810074cb7950(00007ffffffffb18). mm/memory.c:97: bad pmd ffff810074cb7958(00007ffffffffb19). mm/memory.c:97: bad pmd ffff810074cb7960(00007ffffffffb1a). mm/memory.c:97: bad pmd ffff810074cb7968(00007ffffffffb1b). mm/memory.c:97: bad pmd ffff810074cb7970(00007ffffffffb1c). mm/memory.c:97: bad pmd ffff810074cb7978(00007ffffffffb1d). mm/memory.c:97: bad pmd ffff810074cb7980(00007ffffffffb1e). mm/memory.c:97: bad pmd ffff810074cb7988(00007ffffffffb1f). mm/memory.c:97: bad pmd ffff810074cb7990(00007ffffffffb20). mm/memory.c:97: bad pmd ffff810074cb7998(00007ffffffffb21). mm/memory.c:97: bad pmd ffff810074cb79a0(00007ffffffffb22). mm/memory.c:97: bad pmd ffff810074cb79a8(00007ffffffffb23). mm/memory.c:97: bad pmd ffff810074cb79b0(00007ffffffffb24). mm/memory.c:97: bad pmd ffff810074cb79b8(00007ffffffffb25). mm/memory.c:97: bad pmd ffff810074cb79c0(00007ffffffffb26). mm/memory.c:97: bad pmd ffff810074cb79c8(00007ffffffffb27). mm/memory.c:97: bad pmd ffff810074cb79d0(00007ffffffffb28). mm/memory.c:97: bad pmd ffff810074cb79d8(00007ffffffffb29). mm/memory.c:97: bad pmd ffff810074cb79e0(00007ffffffffb2a). mm/memory.c:97: bad pmd ffff810074cb79f0(0000000000000010). mm/memory.c:97: bad pmd ffff810074cb79f8(00000000078bfbff). mm/memory.c:97: bad pmd ffff810074cb7a00(0000000000000006). mm/memory.c:97: bad pmd ffff810074cb7a08(0000000000001000). mm/memory.c:97: bad pmd ffff810074cb7a10(0000000000000011). mm/memory.c:97: bad pmd ffff810074cb7a18(0000000000000064). mm/memory.c:97: bad pmd ffff810074cb7a20(0000000000000003). mm/memory.c:97: bad pmd ffff810074cb7a28(0000000000400040). mm/memory.c:97: bad pmd ffff810074cb7a30(0000000000000004). mm/memory.c:97: bad pmd ffff810074cb7a38(0000000000000038). mm/memory.c:97: bad pmd ffff810074cb7a40(0000000000000005). mm/memory.c:97: bad pmd ffff810074cb7a48(0000000000000008). mm/memory.c:97: bad pmd ffff810074cb7a50(0000000000000007). mm/memory.c:97: bad pmd ffff810074cb7a60(0000000000000008). mm/memory.c:97: bad pmd ffff810074cb7a70(0000000000000009). mm/memory.c:97: bad pmd ffff810074cb7a78(0000000000401480). mm/memory.c:97: bad pmd ffff810074cb7a80(000000000000000b). mm/memory.c:97: bad pmd ffff810074cb7a90(000000000000000c). mm/memory.c:97: bad pmd ffff810074cb7aa0(000000000000000d). mm/memory.c:97: bad pmd ffff810074cb7ab0(000000000000000e). mm/memory.c:97: bad pmd ffff810074cb7ac0(0000000000000017). mm/memory.c:97: bad pmd ffff810074cb7ad0(000000000000000f). mm/memory.c:97: bad pmd ffff810074cb7ad8(00007ffffffffaf9). mm/memory.c:97: bad pmd ffff810074cb7af8(0034365f36387800).
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Hugh DickinsQuoted 10 lines
It does. Well, I needed to restudy
exec_mmapandswitch_mmin detail, and having done so, I agree that the only way you can get through exec_mmap'sactivate_mmwithout fully flushing the cpu's TLB, is if theactive_mmmatches the newly allocated mm (itself impossible since there's a reference on theactive_mm), and the cpu bit is still set incpu_vm_mask- precisely not the case if we went throughleave_mm. Yet I was claiming yourleave_mmfix could flush TLB forexec_mmapwhere it wasn't already done.
Quoted 5 lines
Sorry for letting the neatness of my pmd/stack story blind me to its impossibility, and for wasting your time.
Hugh
-
Any updated information one should know about this before testing?
I'm getting bad pmds in 2.6.11.5; Tyan S2882/dual AMD 246 opterons. The problem only occurs when doing some thread intensive task. I'm going to try and strace/bt and send some information as it occurs. Hardware is almost identical to the setup above.
-Christopher Warner
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 1 line · * Christopher Warner ([email protected]) wrote:
Any updated information one should know about this before testing?
Andi would like to see output from this patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=111461231610952&w=2
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Christopher WarnerQuoted 13 lines · Christopher Warner
Quoted 13 lines
It does. Well, I needed to restudy
exec_mmapandswitch_mmin detail, and having done so, I agree that the only way you can get through exec_mmap'sactivate_mmwithout fully flushing the cpu's TLB, is if theactive_mmmatches the newly allocated mm (itself impossible since there's a reference on theactive_mm), and the cpu bit is still set incpu_vm_mask- precisely not the case if we went throughleave_mm. Yet I was claiming yourleave_mmfix could flush TLB forexec_mmapwhere it wasn't already done.Sorry for letting the neatness of my pmd/stack story blind me to its impossibility, and for wasting your time.
Quoted 6 lines
Quoted 2 lines
Hugh
-Any updated information one should know about this before testing?
I'm getting bad pmds in 2.6.11.5; Tyan S2882/dual AMD 246 opterons.
Datapoint: exactly the same model as my workstation which showed this problem recently.
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 2 lines
Datapoint: exactly the same model as my workstation which showed this problem recently.
FYI
Since all the people who run into this are refusing to test my debugging/test patches (no feedback at all for them so far) I give up on this now.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Actually I am testing your patches. Its just going to take some time. The problem occurs under severe load and I'm in the process of doing load testing this for an inhouse app this week. Soon as i'm able to send debug information I will.
-Christopher Warner
Quoted 11 lines · Andi Kleen
Quoted 2 lines
Datapoint: exactly the same model as my workstation which showed this problem recently.
FYI
Since all the people who run into this are refusing to test my debugging/test patches (no feedback at all for them so far) I give up on this now.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 4 lines · * Christopher Warner ([email protected]) wrote:
Actually I am testing your patches. Its just going to take some time. The problem occurs under severe load and I'm in the process of doing load testing this for an inhouse app this week. Soon as i'm able to send debug information I will.
Same here. I've just never found a way to trigger other than wait.
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 7 lines · Chris Wright
Quoted 4 lines · * Christopher Warner ([email protected]) wrote:
Actually I am testing your patches. Its just going to take some time. The problem occurs under severe load and I'm in the process of doing load testing this for an inhouse app this week. Soon as i'm able to send debug information I will.
Same here. I've just never found a way to trigger other than wait.
*nod*, the current test-kernel update for Fedora also has your debugging patches, but none of the users have hit them (or reported them) yet.
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 7 lines · Dave Jones
Quoted 6 lines · Chris Wright
Quoted 4 lines · * Christopher Warner ([email protected]) wrote:
Actually I am testing your patches. Its just going to take some time. The problem occurs under severe load and I'm in the process of doing load testing this for an inhouse app this week. Soon as i'm able to send debug information I will.
Quoted 5 lines
Quoted 1 line
Same here. I've just never found a way to trigger other than wait.
*nod*, the current test-kernel update for Fedora also has your debugging patches, but none of the users have hit them (or reported them) yet.
The second version with the WARN_ON? If not please update to that one.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 8 lines · Andi Kleen
Quoted 7 lines · Dave Jones
Quoted 6 lines · Chris Wright
Quoted 4 lines · * Christopher Warner ([email protected]) wrote:
Actually I am testing your patches. Its just going to take some time. The problem occurs under severe load and I'm in the process of doing load testing this for an inhouse app this week. Soon as i'm able to send debug information I will.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Same here. I've just never found a way to trigger other than wait.
*nod*, the current test-kernel update for Fedora also has your debugging patches, but none of the users have hit them (or reported them) yet.
The second version with the
WARN_ON? If not please update to that one.
I lost track.. Here's what I included..
mm/memory.c renamed from mm/memory.c~ +2 -0
@@ -94,6 +94,7 @@ static inline void clear_pmd_range(strucif (pmd_none (*pmd))return ;if (unlikely (pmd_bad (*pmd))) { +printk ("%s:%d: " , current->comm, current->pid);pmd_ERROR (*pmd);pmd_clear (pmd);return ;
@@ -113,6 +114,7 @@ static inline void clear_pud_range(strucunsigned long addr = start, next;pmd_t *pmd, *__pmd; +WARN_ON (start == end);if (pud_none (*pud))return ;if (unlikely (pud_bad (*pud))) {
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Christopher Warner2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47). sh:18983: mm/memory.c:99: bad pmd ffff810005974cd8(00007ffffffffe48). sh:18983: mm/memory.c:99: bad pmd ffff810005974ce0(00007ffffffffe49). sh:18983: mm/memory.c:99: bad pmd ffff810005974ce8(00007ffffffffe4a). sh:18983: mm/memory.c:99: bad pmd ffff810005974cf0(00007ffffffffe4b). sh:18983: mm/memory.c:99: bad pmd ffff810005974cf8(00007ffffffffe4c). sh:18983: mm/memory.c:99: bad pmd ffff810005974d00(00007ffffffffe4d). sh:18983: mm/memory.c:99: bad pmd ffff810005974d08(00007ffffffffe4e). sh:18983: mm/memory.c:99: bad pmd ffff810005974d10(00007ffffffffe4f). sh:18983: mm/memory.c:99: bad pmd ffff810005974d18(00007ffffffffe50). sh:18983: mm/memory.c:99: bad pmd ffff810005974d20(00007ffffffffe51). sh:18983: mm/memory.c:99: bad pmd ffff810005974d30(0000000000000010). sh:18983: mm/memory.c:99: bad pmd ffff810005974d38(00000000078bfbff). sh:18983: mm/memory.c:99: bad pmd ffff810005974d40(0000000000000006). sh:18983: mm/memory.c:99: bad pmd ffff810005974d48(0000000000001000). sh:18983: mm/memory.c:99: bad pmd ffff810005974d50(0000000000000011). sh:18983: mm/memory.c:99: bad pmd ffff810005974d58(0000000000000064). sh:18983: mm/memory.c:99: bad pmd ffff810005974d60(0000000000000003). sh:18983: mm/memory.c:99: bad pmd ffff810005974d68(0000000000400040). sh:18983: mm/memory.c:99: bad pmd ffff810005974d70(0000000000000004). sh:18983: mm/memory.c:99: bad pmd ffff810005974d78(0000000000000038). sh:18983: mm/memory.c:99: bad pmd ffff810005974d80(0000000000000005). sh:18983: mm/memory.c:99: bad pmd ffff810005974d88(0000000000000008). sh:18983: mm/memory.c:99: bad pmd ffff810005974d90(0000000000000007). sh:18983: mm/memory.c:99: bad pmd ffff810005974d98(00002aaaaaaab000). sh:18983: mm/memory.c:99: bad pmd ffff810005974da0(0000000000000008). sh:18983: mm/memory.c:99: bad pmd ffff810005974db0(0000000000000009). sh:18983: mm/memory.c:99: bad pmd ffff810005974db8(0000000000413a00). sh:18983: mm/memory.c:99: bad pmd ffff810005974dc0(000000000000000b). sh:18983: mm/memory.c:99: bad pmd ffff810005974dc8(00000000000007d3). sh:18983: mm/memory.c:99: bad pmd ffff810005974dd0(000000000000000c). sh:18983: mm/memory.c:99: bad pmd ffff810005974dd8(00000000000007d3). sh:18983: mm/memory.c:99: bad pmd ffff810005974de0(000000000000000d). sh:18983: mm/memory.c:99: bad pmd ffff810005974df0(000000000000000e). sh:18983: mm/memory.c:99: bad pmd ffff810005974e00(0000000000000017). sh:18983: mm/memory.c:99: bad pmd ffff810005974e10(000000000000000f). sh:18983: mm/memory.c:99: bad pmd ffff810005974e18(00007ffffffffe3a). sh:18983: mm/memory.c:99: bad pmd ffff810005974e38(34365f3638780000)
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 2 lines · * Christopher Warner ([email protected]) wrote:
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
Got a time stamp by any chance (or a clue re: what was going on at the time)?
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Christopher WarnerQuoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 9 lines · Dave Jones
Quoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
It seems to be clear now that it is hardware independent.
I actually got it once now too, but only after 24+h stress test :/
I have a better debugging patch now that I will be testing soon, hopefully that turns something up.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Chris WrightThats from a couple of days ago. The bug used to be easily reproducible and then it just stopped suddenly.
I've thrown the loads of the server machines in question way into the 2000 range with lots of threads, io, cpu usage. I'm really confused as to what exactly it could be. I'm going to try on a couple of different machines this week.
I'm starting to suspect it has something to do with the mobo itself, since I have two or three next to me I'll try on those other machines and then try on a completely different machine/motherboard.
It's really sneaky, so we'll see what happens.
Quoted 9 lines · Chris Wright
Quoted 2 lines · * Christopher Warner ([email protected]) wrote:
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opteronsGot a time stamp by any chance (or a clue re: what was going on at the time)?
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Andi KleenQuoted 11 lines · Andi Kleen
Quoted 9 lines · Dave Jones
Quoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
Quoted 6 lines
It seems to be clear now that it is hardware independent.
I actually got it once now too, but only after 24+h stress test :/
I have a better debugging patch now that I will be testing soon, hopefully that turns something up.
Ok, I'm respinning the Fedora update kernel today for other reasons, if you have that patch in time, I'll toss it in too.
Though as yet, no further reports from our users.
Dave
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 14 lines · Dave Jones
Quoted 13 lines · Andi Kleen
Quoted 9 lines · Dave Jones
Quoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
It seems to be clear now that it is hardware independent.
Quoted 7 lines
Quoted 4 lines
I actually got it once now too, but only after 24+h stress test :/
I have a better debugging patch now that I will be testing soon, hopefully that turns something up.
Ok, I'm respinning the Fedora update kernel today for other reasons, if you have that patch in time, I'll toss it in too.
The patch has considerable overhead, probably not good idea for a production rpm.
-Andi
Re: x86-64 bad pmds in 2.6.11.6 II
Quoted 17 lines · * Andi Kleen ([email protected]) wrote:
Quoted 16 lines · Dave Jones
Quoted 15 lines · Andi Kleen
Quoted 9 lines · Dave Jones
Quoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
It seems to be clear now that it is hardware independent.
I actually got it once now too, but only after 24+h stress test :/
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I have a better debugging patch now that I will be testing soon, hopefully that turns something up.
Ok, I'm respinning the Fedora update kernel today for other reasons, if you have that patch in time, I'll toss it in too.
The patch has considerable overhead, probably not good idea for a production rpm.
I don't mind running it here. I've triggered it once, and not hit the WARN_ON(start == end). Current was "sh", not that helpful.
thanks,
-chris
Re: x86-64 bad pmds in 2.6.11.6 II
In reply to Dave JonesQuoted 16 lines · Dave Jones
Quoted 15 lines · Andi Kleen
Quoted 9 lines · Dave Jones
Quoted 4 lines · Christopher Warner
2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons
sh:18983:mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). sh:18983:mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).That's the 3rd or 4th time I've seen this reported on this hardware. It's not exclusive to it, but it does seem more susceptible for some reason. Spooky.
It seems to be clear now that it is hardware independent.
I actually got it once now too, but only after 24+h stress test :/
Quoted 7 lines
Quoted 2 lines
I have a better debugging patch now that I will be testing soon, hopefully that turns something up.
Ok, I'm respinning the Fedora update kernel today for other reasons, if you have that patch in time, I'll toss it in too.
Though as yet, no further reports from our users.
Here's the new patch. However it costs some memory bloat because I added a new field to struct page
-Andi
mm/page_alloc.c +6 -1
@@ -912,6 +912,7 @@ nopage:return NULL ; got_pg:zone_statistics (zonelist, z); + page->freer = (void *)-1ULL ;return page; }
@@ -962,13 +963,16 @@ void __pagevec_free(struct pagevec *pvec {int i =pagevec_count (pvec); -while (--i >=0 ) +while (--i >=0 ) { + pvec->pages[i]->freer = (void *)__builtin_return_address (0 );free_hot_cold_page (pvec->pages[i], pvec->cold); + } } fastcall void __free_pages(struct page *page, unsigned int order) {if (!PageReserved (page) &&put_page_testzero (page)) { + page->freer = (void *)__builtin_return_address (0 );if (order ==0 )free_hot_page (page);else
@@ -1595,6 +1599,7 @@ void __init memmap_init_zone(unsigned lostruct page *page;for (page = start; page < (start + size); page++) { + page->freer =NULL ;set_page_zone (page,NODEZONE (nid, zone));set_page_count (page,0 );reset_page_mapcount (page);
include/linux/mm.h +1 -0
@@ -261,6 +261,7 @@ struct page {void *virtual;/* Kernel virtual address (NULL if not kmapped, ie. highmem) */ #endif /* WANT_PAGE_VIRTUAL */ +void *freer; };/*
mm/memory.c +9 -0
@@ -48,6 +48,7 @@#include <linux/rmap.h> #include <linux/module.h> #include <linux/init.h> +#include <linux/kallsyms.h> #include <asm/pgalloc.h> #include <asm/uaccess.h>
@@ -108,6 +109,14 @@ static inline void clear_pmd_range(struc pmd =pmd_offset (pud, addr); + { +struct page *p =virt_to_page (pmd); +if (page_count (p) <1 ) { +printk ("%s:%d free pmd %lx " , current->comm, current->pid, addr); +print_symbol ("freed by %s\n" , (unsigned long )p->freer); + } + } +/* Only free fully aligned ranges */ if (!((addr | end) & ~PUD_MASK )) empty_pmd = pmd;
Re: x86-64 bad pmds in 2.6.11.6 II
CW = Christopher Warner
CW>>>>> 2.6.11.5 kernel,
CW>>>>> Tyan S2882/dual AMD 246 opterons
CW>>>>> sh:18983: mm/memory.c:99: bad pmd
ffff810005974cc8(00007ffffffffe46).
CW>>>>> sh:18983: mm/memory.c:99: bad pmd
ffff810005974cd0(00007ffffffffe47).
DJ = Dave Jones
DJ>>>> That's the 3rd or 4th time I've seen this DJ>>>> reported on this hardware.
DJ>>>> It's not exclusive to it, but it does seem more DJ>>>> susceptible for some reason. Spooky.
AK = Andi Kleen wrote:
AK>>> It seems to be clear now that it is hardware AK>>> independent.
AK>>>
AK>>> I actually got it once now too, but only after AK>>> 24+h stress test :/
AK>>>
AK>>> I have a better debugging patch now that I will be AK>>> testing soon, hopefully that turns something up.
DJ = Dave Jones
DJ>> Ok, I'm respinning the Fedora update kernel today DJ>> for other reasons, if you have that patch in time, DJ>> I'll toss it in too.
DJ>>
DJ>> Though as yet, no further reports from our users.
AK = Andi Kleen
AK> Here's the new patch. However it costs some memory AK> bloat because I added a new field to struct page
I posted some information on the fedora-list concerning my experience with this problem. I am using a Tyan S2885/dual 244 Opterons. For HW and driver details see:
https://www.redhat.com/archives/fedora-list/2005-May/msg01690.html
I have been using Dave's FC3 test kernel (2.6.11-1.24_FC3smp) for a little over a day and have been unable to generate the problem with the computer under a larger than normal load.
Prior to May 12, I had been seeing the problem very regularly. It started around April 14. I believe this is about the time I first started using the 2.6.11-1.14_FC3smp kernel. I remember I had to get a BIOS upgrade from Tyan (http://www.tyan.com/support/html/b_s2885.html
unfortunately a Beta release) to get my network back once I started using the new kernel. After that the memory.c messages started showing up. I though it might be BIOS related (see the note on the referenced Tyan page), but when I saw Christopher's post I thought maybe it was the kernel because his MOBO doesn't have a Beta BIOS release. I googled and found this thread, and subscribed to this list.
I have "memory.c:97 bad pmd" entries in my /var/log/messages files going back to April 14. The only days I don't have them are April 20, 22, 23, 24 and April 29 (22, 23, and 24 are a weekend with less activity). I have had them every day in May until I installed Dave's test kernel.
I am very computer literate, but not a kernel developer. I hope I didn't offend you guys by posting here. I would be willing to be your guinea pig for testing. Currently I am unable to reproduce the problem. If I am able to reproduce the problem, would you prefer I post here or on the fedora list?
Thanks for all of your efforts,
Pete
Re: x86-64 bad pmds in 2.6.11.6 II
CW = Christopher Warner
CW>>>>>> 2.6.11.5 kernel,
CW>>>>>> Tyan S2882/dual AMD 246 opterons
CW>>>>>> sh:18983: mm/memory.c:99: bad pmd
CW>>>>>> ffff810005974cc8(00007ffffffffe46).
CW>>>>>> sh:18983: mm/memory.c:99: bad pmd
CW>>>>>> ffff810005974cd0(00007ffffffffe47).
DJ = Dave Jones
DJ>>>>> That's the 3rd or 4th time I've seen this DJ>>>>> reported on this hardware.
DJ>>>>> It's not exclusive to it, but it does seem more DJ>>>>> susceptible for some reason. Spooky.
AK = Andi Kleen wrote:
AK>>>> It seems to be clear now that it is hardware AK>>>> independent.
AK>>>>
AK>>>> I actually got it once now too, but only after AK>>>> 24+h stress test :/
AK>>>>
AK>>>> I have a better debugging patch now that I will be AK>>>> testing soon, hopefully that turns something up.
DJ = Dave Jones
DJ>>> Ok, I'm respinning the Fedora update kernel today DJ>>> for other reasons, if you have that patch in time, DJ>>> I'll toss it in too.
DJ>>>
DJ>>> Though as yet, no further reports from our users.
PJS = Peter J. Stieber
PJS> I posted some information on the fedora-list
PJS> concerning my experience with this problem. I
PJS> am using a Tyan S2885/dual 244 Opterons. For HW and PJS> driver details see:
PJS> https://www.redhat.com/archives/fedora-list/2005-May/msg01690.html
PJS> I have been using Dave's FC3 test kernel
PJS> (2.6.11-1.24_FC3smp) for a little over a day and
PJS> have been unable to generate the problem with the
PJS> computer under a larger than normal load.
PJS>
PJS> Prior to May 12, I had been seeing the problem very PJS> regularly. It started around April 14. I believe this PJS> is about the time I first started using the
PJS> 2.6.11-1.14_FC3smp kernel. I remember I had to get a PJS> BIOS upgrade from Tyan
PJS> (http://www.tyan.com/support/html/b_s2885.html
PJS> unfortunately a Beta release) to get my network back PJS> once I started using the new kernel. After that the PJS> memory.c messages started showing up. I though it
PJS> might be BIOS related (see the note on the referenced PJS> Tyan page), but when I saw Christopher's post I
PJS> thought maybe it was the kernel because his MOBO
PJS> doesn't have a Beta BIOS release. I googled and
PJS> found this thread, and subscribed to this list.
PJS>
PJS> I have "memory.c:97 bad pmd" entries in my
PJS> /var/log/messages files going back to April 14.
PJS> The only days I don't have them are April 20, 22, 23, PJS> 24 and April 29 (22, 23, and 24 are a weekend with
PJS> less activity). I have had them every day in May
PJS> until I installed Dave's test kernel.
PJS>
PJS> I am very computer literate, but not a kernel
PJS> developer. I hope I didn't offend you guys by
PJS> posting here. I would be willing to be your guinea
PJS> pig for testing. Currently I am unable to reproduce PJS> the problem. If I am able to reproduce the problem, PJS> would you prefer I post here or on the
PJS> fedora list?
PJS>
PJS> Thanks for all of your efforts,
My Tyan S2885 just exhibited the problem with Dave's test FC3 kernel (2.6.11-1.24_FC3smp):
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d008(0000000000000008). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d018(0000000000000009). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d020(0000000000401b80). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d028(000000000000000b). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d030(00000000000001f4). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d038(000000000000000c). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d040(00000000000001f4). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d048(000000000000000d). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d050(00000000000001f7). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d058(000000000000000e). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d060(00000000000001f7). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d068(0000000000000017). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d078(000000000000000f). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d080(00007ffffffff0a4). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd ffff81005856d0a0(5f36387800000000). May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d0a8(0000000000003436).
It died while running configure. It was attempting to use gcc to make sure C compiling was possible. In the past, if I run the script a second time it works.
Sorry for my ignorance, but what other information do you need? Pete