PROBLEM: BUG: unable to handle kernel paging request at ffff81013a612000
Hi,
Please CC me with any replies as I'm not subscribed to the list.
I'm running a vanilla 2.6.25.4 kernel.
I apparently hit a bug in the kernel.
When the problem starts my system becomes very unstable, many programs crash others I can barely shut down correctly and a reboot fails.
I've hit the problem once trying to do "echo disk > /sys/power/state" and another time while writing do the disk inside a vmware guest. And a couple of times it happened randomly, e.g. while converting some files to mp3's and writing them to a usb device.
(More details about my system at the end of the mail) dmesg output:
BUG: unable to handle kernel paging request at ffff81013a612000 IP: [<ffffffff8027391b>] __slab_alloc+0x24a/0x4f8 PGD 8063 PUD d063 PMD 800040013a6001e3 Oops: 000b [1] SMP CPU 0 Modules linked in: vmnet(PF) vmmon(PF) vmblock pwc compat_ioctl32 videodev v4l1_compat snd_usb_audio snd_usb_lib snd_rawmidi snd_hwdep it87 hwmon_vid xts gf128mul dm_crypt bridge llc tun vfat fat usb_storage nvidia(P) i2c_i801 snd_hda_intel Pid: 15862, comm: pdflush Tainted: PF 2.6.25.4 #5 RIP: 0010:[<ffffffff8027391b>] [<ffffffff8027391b>] __slab_alloc+0x24a/0x4f8 RSP: 0018:ffff81006b28b860 EFLAGS: 00010046 RAX: 0000000000000000 RBX: ffff81013a612000 RCX: 000000000000000e RDX: ffff81013a612000 RSI: ffffe200044c53f0 RDI: ffffffff8061cea0 RBP: ffffffff8061cea0 R08: ffffffff805e4e6a R09: 0000000000000989 R10: 0000000000000000 R11: 0000000000000000 R12: ffff81013a612000 R13: ffffe200044c53f0 R14: ffff81013a612000 R15: 0000000000011200 FS: 0000000000000000(0000) GS:ffffffff80610000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: ffff81013a612000 CR3: 000000014a7d8000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000002c56 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process pdflush (pid: 15862, threadinfo ffff81006b28a000, task ffff8100a59ef6b0) Stack: ffffffff80257dae ffffffff2ec56000 0000000000011200 0000000000000206 ffff8100010065d0 0000000000011200 0000000000011210 ffff81006b28b8e0 ffff81022e6444b0 ffffffff80273df8 0000000000000010 ffff81022e644480 Call Trace: [<ffffffff80257dae>] ? mempool_alloc+0x50/0x104 [<ffffffff80273df8>] ? kmem_cache_alloc+0x35/0x65 [<ffffffff80257dae>] ? mempool_alloc+0x50/0x104 [<ffffffff80297b24>] ? bio_alloc_bioset+0x95/0xea [<ffffffff8040c7be>] ? clone_bio+0x37/0x7a [<ffffffff8040d54d>] ? __split_bio+0x13d/0x378 [<ffffffff803157a1>] ? xfs_convert_page+0x2ed/0x30f [<ffffffff8040dccd>] ? dm_request+0x104/0x126 [<ffffffff80330b92>] ? generic_make_request+0x1e1/0x21c [<ffffffff80331f97>] ? submit_bio+0xc8/0xcf [<ffffffff803150d6>] ? xfs_submit_ioend_bio+0x1e/0x27 [<ffffffff8031519b>] ? xfs_submit_ioend+0xa7/0xc6 [<ffffffff80315f3c>] ? xfs_page_state_convert+0x4f1/0x540 [<ffffffff803160d4>] ? xfs_vm_writepage+0xa5/0xdd [<ffffffff8025ad57>] ? __writepage+0xa/0x23 [<ffffffff8025b1d6>] ? write_cache_pages+0x182/0x2cb [<ffffffff8025ad4d>] ? __writepage+0x0/0x23 [<ffffffff8025b35b>] ? do_writepages+0x20/0x2d [<ffffffff802907e9>] ? __writeback_single_inode+0x140/0x294 [<ffffffff8040eef9>] ? dm_table_any_congested+0x46/0x6c [<ffffffff80290cc8>] ? sync_sb_inodes+0x1c2/0x2ad [<ffffffff802910fd>] ? writeback_inodes+0x67/0xb8 [<ffffffff8025bbc5>] ? wb_kupdate+0xa0/0x113 [<ffffffff8025bf42>] ? pdflush+0x0/0x1e5 [<ffffffff8025c081>] ? pdflush+0x13f/0x1e5 [<ffffffff8025bb25>] ? wb_kupdate+0x0/0x113 [<ffffffff8023dcf5>] ? kthread+0x47/0x76 [<ffffffff8022810c>] ? schedule_tail+0x28/0x5c [<ffffffff8020bd58>] ? child_rip+0xa/0x12 [<ffffffff8021b3b7>] ? lapic_next_event+0x0/0xa [<ffffffff8023dcae>] ? kthread+0x0/0x76 [<ffffffff8020bd4e>] ? child_rip+0x0/0x12 Code: 00 be 5a 00 00 00 4c 89 f7 48 d3 e2 e8 8f c7 0c 00 4d 89 f4 4c 89 f3 eb 20 4c 89 e2 4c 89 ee 48 89 ef e8 fa ec ff ff 48 63 45 10 <49> 89 1c 04 48 63 45 08 49 89 dc 48 01 c3 8b 45 08 0f af 45 50 RIP [<ffffffff8027391b>] __slab_alloc+0x24a/0x4f8 RSP <ffff81006b28b860> CR2: ffff81013a612000 ---[ end trace bf8cedeb8138fc52 ]--- BUG: unable to handle kernel paging request at ffff81013a613000 IP: [<ffffffff8033f947>] clear_page_c+0x7/0x10 PGD 8063 PUD d063 PMD 800040013a6001e3 Oops: 000b [2] SMP CPU 3 Modules linked in: vmnet(PF) vmmon(PF) vmblock pwc compat_ioctl32 videodev v4l1_compat snd_usb_audio snd_usb_lib snd_rawmidi snd_hwdep it87 hwmon_vid xts gf128mul dm_crypt bridge llc tun vfat fat usb_storage nvidia(P) i2c_i801 snd_hda_intel Pid: 18392, comm: bash Tainted: PF D 2.6.25.4 #5 RIP: 0010:[<ffffffff8033f947>] [<ffffffff8033f947>] clear_page_c+0x7/0x10 RSP: 0000:ffff8101348f3d60 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffe200044c5428 RCX: 0000000000000200 RDX: 0000000000000000 RSI: 00000000ffffffff RDI: ffff81013a613000 RBP: 00000000044c5428 R08: 0000000000000000 R09: 0000000000000870 R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000 R13: ffff810000000000 R14: 6db6db6db6db6db7 R15: 00000000001280d2 FS: 00007f782f83e6f0(0000) GS:ffff81022fc0e680(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff81013a613000 CR3: 0000000186308000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process bash (pid: 18392, threadinfo ffff8101348f2000, task ffff81006b1b6f20) Stack: ffffffff8025a269 0000000000000044 00000000805e5ae8 0000000000000001 ffffffff80664340 0010000000000001 0000000000000000 ffffffff805e4f40 0000000000000002 0000000000000044 ffffffff805e5b18 0000000100000002 Call Trace: [<ffffffff8025a269>] ? get_page_from_freelist+0x361/0x3f6 [<ffffffff8025a54e>] ? __alloc_pages+0xa4/0x32a [<ffffffff80263d86>] ? handle_mm_fault+0x230/0x6ab [<ffffffff802660ac>] ? __vma_link+0x42/0x4b [<ffffffff8021f667>] ? do_page_fault+0x36d/0x72e [<ffffffff8033df15>] ? __up_write+0x21/0x110 [<ffffffff804e40e9>] ? error_exit+0x0/0x51 Code: af 74 08 48 83 ef 08 48 0f bc 07 48 29 d7 48 c1 e7 03 48 01 f8 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 <f3> 48 ab c3 0f 1f 44 00 00 eb ee 0f 1f 84 00 00 00 00 00 0f 1f RIP [<ffffffff8033f947>] clear_page_c+0x7/0x10 RSP <ffff8101348f3d60> CR2: ffff81013a613000 ---[ end trace bf8cedeb8138fc52 ]---
note: bash[18392] exited with preempt_count 1
-------------------------------------------------
linux # scripts/ver_linux
If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
Linux simon 2.6.25.4 #6 SMP Sun May 25 22:17:29 CEST 2008 x86_64 Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz GenuineIntel GNU/Linux
Gnu C 4.2.4 Gnu make 3.81 binutils 2.18 util-linux 2.13.1.1 mount 2.13.1.1 module-init-tools 3.4 e2fsprogs 1.40.8 xfsprogs 2.9.7 Linux C Library 2.6.1 Dynamic linker (ldd) 2.6.1 Procps 3.2.7 Net-tools 1.60 Kbd 1.13 Sh-utils 6.10 udev 119 Modules Loaded cifs bridge llc tun pwc i2c_i801 snd_usb_audio compat_ioctl32 videodev snd_usb_lib v4l1_compat snd_rawmidi snd_hwdep snd_hda_intel nvidia
linux # cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-006f : keyboard
0070-0077 : rtc
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0290-029f : pnp 00:01
0290-0294 : pnp 00:01
03c0-03df : vesafb
03f8-03ff : serial
0400-047f : 0000:00:1f.0
0400-0403 : ACPI PM1a_EVT_BLK
0404-0405 : ACPI PM1a_CNT_BLK
0408-040b : ACPI PM_TMR
0410-0415 : ACPI CPU throttle
0420-042f : ACPI GPE0_BLK
0480-04bf : 0000:00:1f.0
04d0-04d1 : pnp 00:01
0500-051f : 0000:00:1f.3
0500-051f : i801_smbus
0800-087f : pnp 00:01
0880-088f : pnp 00:01
0cf8-0cff : PCI conf1
a000-afff : PCI Bus #05
b000-bfff : PCI Bus #02
c000-cfff : PCI Bus #03
c000-c007 : 0000:03:00.1
c000-c007 : pata_jmicron
c100-c103 : 0000:03:00.1
c100-c103 : pata_jmicron
c200-c207 : 0000:03:00.1
c200-c207 : pata_jmicron
c300-c303 : 0000:03:00.1
c300-c303 : pata_jmicron
c400-c40f : 0000:03:00.1
c400-c40f : pata_jmicron
d000-dfff : PCI Bus #04
d000-d0ff : 0000:04:00.0
d000-d0ff : r8169e000-e01f : 0000:00:1a.2
e000-e01f : uhci_hcd
e100-e11f : 0000:00:1a.0
e100-e11f : uhci_hcd
e200-e21f : 0000:00:1a.1
e200-e21f : uhci_hcd
e300-e31f : 0000:00:1d.0
e300-e31f : uhci_hcd
e400-e41f : 0000:00:1d.1
e400-e41f : uhci_hcd
e500-e51f : 0000:00:1d.2
e500-e51f : uhci_hcd
e600-e607 : 0000:00:1f.2
e600-e607 : ahci
e700-e703 : 0000:00:1f.2
e700-e703 : ahci
e800-e807 : 0000:00:1f.2
e800-e807 : ahci
e900-e903 : 0000:00:1f.2
e900-e903 : ahci
ea00-ea1f : 0000:00:1f.2
ea00-ea1f : ahci
linux # cat /proc/iomem
00000000-0009e7ff : System RAM
0009f800-0009ffff : reserved
000d5000-000d7fff : pnp 00:0b
000e0000-000effff : pnp 00:0b
000f0000-000fffff : reserved
00100000-cfedffff : System RAM
00200000-004e6b4e : Kernel code
004e6b4f-0060fe27 : Kernel data
0066a000-006a73cf : Kernel bss
cfee0000-cfee2fff : ACPI Non-volatile Storage
cfee3000-cfeeffff : ACPI Tables
cfef0000-cfefffff : reserved
d0000000-dfffffff : PCI Bus #01
d0000000-dfffffff : 0000:01:00.0
d0000000-dfffffff : vesafb
e0000000-e7ffffff : PCI Bus #01
e0000000-e3ffffff : 0000:01:00.0
e0000000-e3ffffff : nvidia e4000000-e4ffffff : 0000:01:00.0
e8000000-ebffffff : PCI MMCONFIG 0
e8000000-ebffffff : reserved
ec000000-edffffff : PCI Bus #04
ed000000-ed000fff : 0000:04:00.0
ed000000-ed000fff : r8169
ee000000-ee0fffff : PCI Bus #03
ee000000-ee001fff : 0000:03:00.0
ee000000-ee001fff : ahciee100000-ee103fff : 0000:00:1b.0
ee100000-ee103fff : ICH HD audio
ee104000-ee1043ff : 0000:00:1a.7
ee104000-ee1043ff : ehci_hcd
ee105000-ee1053ff : 0000:00:1d.7
ee105000-ee1053ff : ehci_hcd
ee106000-ee1067ff : 0000:00:1f.2
ee106000-ee1067ff : ahci
ee107000-ee1070ff : 0000:00:1f.3
fec00000-fec00fff : IOAPIC 0
fec00000-fec00fff : pnp 00:0b
fed00000-fed003ff : HPET 0
fed10000-fed1dfff : pnp 00:0b
fed20000-fed8ffff : pnp 00:0b
fee00000-fee00fff : Local APIC
ffb00000-ffb7ffff : pnp 00:0b
fff00000-ffffffff : pnp 00:0b
100000000-22fffffff : System RAM
Should you need any more information please let me know.
Signature · 3 lines
regards, Georges Toth
Hi,
Quoted 6 lines · Georges Toth
Hi,
Please CC me with any replies as I'm not subscribed to the list.
I'm running a vanilla 2.6.25.4 kernel.
Quoted 9 lines
I apparently hit a bug in the kernel.
When the problem starts my system becomes very unstable, many programs crash others I can barely shut down correctly and a reboot fails.I've hit the problem once trying to do "echo disk >
/sys/power/state" and another time while writing do the disk inside a vmware guest. And a couple of times it happened randomly, e.g. while converting some files to mp3's and writing them to a usb device.
Thanks for the report.
Quoted 10 lines
BUG: unable to handle kernel paging request at ffff81013a612000 IP: [<ffffffff8027391b>] __slab_alloc+0x24a/0x4f8 PGD 8063 PUD d063 PMD 800040013a6001e3 Oops: 000b [1] SMP CPU 0 Modules linked in: vmnet(PF) vmmon(PF) vmblock pwc compat_ioctl32 videodev
v4l1_compatsnd_usb_audiosnd_usb_libsnd_rawmidisnd_hwdepit87hwmon_vidxts gf128muldm_cryptbridge llc tun vfat fatusb_storagenvidia(P)i2c_i801snd_hda_intel
Pid: 15862, comm: pdflush Tainted: PF 2.6.25.4 #5
It seems that you have three proprietary modules loaded: vmnet, vmmon, and nvidia (identified by the (P) suffix in module list above). Especially the vmnet and vmmon modules which have been loaded forcibly (the F suffix).
This means that it will be very hard for the mainline developers to debug your problem. Most likely your report will be ignored.
I suggest starting your system without loading these kernel modules in the first place. Note: It is important that the modules are not loaded in the first place. Simply unloading them won't work, as the damage may have been done already. If you can reproduce the problem when these modules have not been loaded, then I am sure it will be inspected properly.
If you cannot reproduce the problem, however, I would try to narrow down the problem by loading only one or the other of the modules, then contact the vendor(s) of the offending modules.
The easiest way to prevent a module from loading at boot is to remove the corresponding .ko file from /lib/modules/<kernel-version>/. (But keep the original file if you want to move it back later.)
Good luck.
Vegard
Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation." -- E. W. Dijkstra, EWD1036
Hi,
Quoted 15 lines · Vegard Nossum
It seems that you have three proprietary modules loaded: vmnet, vmmon, and nvidia (identified by the (P) suffix in module list above). Especially the vmnet and vmmon modules which have been loaded forcibly (the F suffix).
This means that it will be very hard for the mainline developers to debug your problem. Most likely your report will be ignored.
I suggest starting your system without loading these kernel modules in the first place. Note: It is important that the modules are not loaded in the first place. Simply unloading them won't work, as the damage may have been done already. If you can reproduce the problem when these modules have not been loaded, then I am sure it will be inspected properly.
Quoted 8 lines
If you cannot reproduce the problem, however, I would try to narrow down the problem by loading only one or the other of the modules, then contact the vendor(s) of the offending modules.
The easiest way to prevent a module from loading at boot is to remove the corresponding .ko file from /lib/modules/<kernel-version>/. (But keep the original file if you want to move it back later.)
As suggested I've removed all proprietary modules and am using the nv xorg module for now. Should that bug occur again, I'll let you know :-).
Thanks
Signature · 3 lines
regards, Georges Toth