PROBLEM: Oops when using dm-crypt with dpt_i2o
Hello everybody,
if I mount a dm-crypt-encrypted Harddisk/RAID-Array on an Adaptec 3200S, and produce load (for example with bonnie++) the kernel crashes.
This is always reproducible. To reproduce simply mount an dm-crypt encrypted device and produce load -- I always used bonnie++.
Unencrypted volumes work fine. Every kind of load I threw at the System worked fine.
I would be very glad, if some kind of solution for this can be found.
Thanks!
The oopses:
#v+
BUG: unable to handle kernel paging request at virtual address 14001000 printing eip: *pde = 00000000 Oops: 0000 [#1] Modules linked in: dm_crypt dm_mod capability commoncap ipv6 pcspkr
ppdev lp parport_pc parport rtc_cmos ppp_generic rtc_core rtc_lib i2c_viapro i2o_core i2c_core via_ircc irda crc_ccitt sg shpchp pci_hotplug evdev thermal processor fan button battery ac slhc r8169 usbcore jfs sr_mod cdrom sd_mod sym53c8xx dpt_i2o ata_generic pata_via libata
CPU: 0 EIP: 0060:[<14001000>] Not tainted VLI EFLAGS: 00010082 (2.6.22.1 #2)
EIP is at 0x14001000
eax: 00000000 ebx: 14001000 ecx: 00000000 edx: 0002bc00
esi: 5b51c000 edi: 14001000 ebp: 5b51b000 esp: dfe21f08
ds: 007b es: 007b fs: 0000 gs: 0000 ss: 0068
Process kblockd/0 (pid: 31, ti=dfe20000 task=c1cfca50 task.ti=dfe20000)
Stack: 5b51a000 14001000 5b519000 14001000 5b518000 d4001000 5b517000
00000086c1d0b534 c1d0b424 c1d0b490 c1da7834 c1da7834 c1da7834 00000000 07000000 c022aac5 c1da78b8 c022b696 c022c762 00000046 c1d0b534 c1d0b424 c1d0b490
Call Trace: [<c022aac5>] __generic_unplug_device+0x25/0x30 [<c022b696>] generic_unplug_device+0x6/0x10 [<c022c762>] blk_unplug_work+0x42/0xa0 [<c022c720>] blk_unplug_work+0x0/0xa0 [<c012a307>] run_workqueue+0x97/0x110 [<c012aa00>] worker_thread+0x0/0xe0 [<c012aa7d>] worker_thread+0x7d/0xe0 [<c012d5e0>] autoremove_wake_function+0x0/0x40 [<c012aa00>] worker_thread+0x0/0xe0 [<c012d292>] kthread+0x42/0x70 [<c012d250>] kthread+0x0/0x70 [<c01041df>] kernel_thread_helper+0x7/0x18 ======================= Code: Bad EIP value.
EIP: [<14001000>] 0x14001000 SS:ESP 0068:dfe21f08
BUG: unable to handle kernel paging request at virtual address 14001000 printing eip: 14001000 *pde = 00000000 Oops: 0000 [#2] Modules linked in: dm_crypt dm_mod capability commoncap ipv6 pcspkr
ppdev lp parport_pc parport rtc_cmos ppp_generic rtc_core rtc_lib i2c_viapro i2o_core i2c_core via_ircc irda crc_ccitt sg shpchp pci_hotplug evdev thermal processor fan button battery ac slhc r8169 usbcore jfs sr_mod cdrom sd_mod sym53c8xx dpt_i2o ata_generic pata_via libata
CPU: 0 EIP: 0060:[<14001000>] Not tainted VLI EFLAGS: 00010086 (2.6.22.1 #2)
EIP is at 0x14001000
eax: 00000000 ebx: 14001000 ecx: 00000000 edx: 0002ba00
esi: 5b4dc000 edi: 14001000 ebp: 5b4db000 esp: dfe21ca4
ds: 007b es: 007b fs: 0000 gs: 0000 ss: 0068
Process kblockd/0 (pid: 31, ti=dfe20000 task=c1cfca50 task.ti=dfe20000)
Stack: 5b4da000 14001000 5b4d9000 14001000 5b4d8000 d4001000 5b4d7000
dfe21cd8c1d0b534 c1d0b424 c1d0b490 c1da7834 c1da7834 00000296 c1d0b490 00000286 c022b823 c1da43c0 c1da7834 c0299030 f7e5c070 00000000 00000001 c02991f4
Call Trace: [<c022b823>] blk_run_queue+0x33/0x70 [<c0299030>] scsi_next_command+0x30/0x50 [<c02991f4>] scsi_end_request+0x94/0xc0 [<c02992fa>] scsi_io_completion+0x8a/0x3e0 [<f884679b>] sd_rw_intr+0x2b/0x220 [sd_mod] [<c0299b7c>] scsi_softirq_done+0x2c/0x100 [<c0295037>] scsi_finish_command+0x47/0x60 [<c022b73a>] blk_done_softirq+0x5a/0x70 [<c0121002>] __do_softirq+0x42/0x90 [<c0121077>] do_softirq+0x27/0x30 [<c012134a>] irq_exit+0x5a/0x60 [<c0105bf9>] do_IRQ+0x39/0x70 [<c01040ef>] common_interrupt+0x23/0x28 [<c011f553>] do_exit+0x73/0x740 [<c0104af7>] die+0x1e7/0x1f0 [<c0114898>] do_page_fault+0x188/0x5e0 [<c0114710>] do_page_fault+0x0/0x5e0 [<c03403aa>] error_code+0x6a/0x70 [<c022aac5>] __generic_unplug_device+0x25/0x30 [<c022b696>] generic_unplug_device+0x6/0x10 [<c022c762>] blk_unplug_work+0x42/0xa0 [<c022c720>] blk_unplug_work+0x0/0xa0 [<c012a307>] run_workqueue+0x97/0x110 [<c012aa00>] worker_thread+0x0/0xe0 [<c012aa7d>] worker_thread+0x7d/0xe0 [<c012d5e0>] autoremove_wake_function+0x0/0x40 [<c012aa00>] worker_thread+0x0/0xe0 [<c012d292>] kthread+0x42/0x70 [<c012d250>] kthread+0x0/0x70 [<c01041df>] kernel_thread_helper+0x7/0x18 ======================= Code: Bad EIP value. EIP: [<14001000>] 0x14001000 SS:ESP 0068:dfe21ca4 Kernel panic - not syncing: Fatal exception in interrupt
#v-
Output from ver_linux:
#v+
Linux beehive 2.6.22.1 #2 Mon Jul 30 11:52:21 CEST 2007 i686 AMD Athlon(tm) Processor AuthenticAMD GNU/Linux
Gnu C 4.2.1 Gnu make 3.81 binutils 2.17 util-linux 2.12r mount 2.12r module-init-tools 3.2.2 e2fsprogs 1.40.2 jfsutils 1.1.11 reiserfsprogs 3.6.20 xfsprogs 2.9.3 pcmciautils 014 Linux C Library > libc.2.6 Dynamic linker (ldd) 2.6 Linux C++ Library so.6.0 Procps 3.2.7 Net-tools 1.60 Kbd 1.12 Sh-utils 6.9 udev 113 wireless-tools 29 Modules Loaded capability commoncap ipv6 ppdev lp parport_pc
parport i2c_viapro i2c_core via_ircc irda crc_ccitt pcspkr rtc_cmos ppp_generic rtc_core rtc_lib sg shpchp pci_hotplug i2o_core evdev thermal processor fan button battery ac slhc r8169 usbcore jfs sr_mod cdrom sd_mod sym53c8xx dpt_i2o ata_generic pata_via libata #v-
/proc/cpuinfo:
#v+
processor : 0vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 4
cpu MHz : 1333.120
cache size : 256 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat
pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 2668.03
clflush size : 32
#v-
Thank you very much,
Signature · 1 line
Philipp Tölke
Quoted 5 lines · On Wed, 01 Aug 2007 20:45:08 +0200 Philipp T__lke <[email protected]> wrote:
Hello everybody,
if I mount a dm-crypt-encrypted Harddisk/RAID-Array on an Adaptec 3200S, and produce load (for example with bonnie++) the kernel crashes.
Quoted 8 lines
This is always reproducible. To reproduce simply mount an dm-crypt encrypted device and produce load -- I always used bonnie++.
Unencrypted volumes work fine. Every kind of load I threw at the System worked fine.
I would be very glad, if some kind of solution for this can be found.
Gee we get a lot of bug reports involving dm-crypt.
Quoted 43 lines
The oopses:
#v+
BUG: unable to handle kernel paging request at virtual address 14001000 printing eip: *pde = 00000000 Oops: 0000 [#1] Modules linked in: dm_crypt dm_mod capability commoncap ipv6 pcspkrppdev lp
parport_pcparportrtc_cmosppp_genericrtc_corertc_libi2c_viaproi2o_corei2c_corevia_irccirdacrc_ccittsg shpchppci_hotplugevdev thermal processor fan button battery ac slhc r8169 usbcore jfssr_modcdromsd_modsym53c8xxdpt_i2oata_genericpata_vialibataCPU: 0 EIP: 0060:[<14001000>] Not tainted VLI EFLAGS: 00010082 (2.6.22.1 #2)EIP is at
0x14001000
eax: 00000000 ebx: 14001000 ecx: 00000000 edx:0002bc00
esi:5b51c000edi: 14001000 ebp:5b51b000esp:dfe21f08
ds: 007b es: 007b fs: 0000 gs: 0000 ss: 0068
Process kblockd/0 (pid: 31, ti=dfe20000 task=c1cfca50 task.ti=dfe20000)Stack: 5b51a000 14001000 5b519000 14001000 5b518000 d4001000 5b51700000000086
c1d0b534c1d0b424c1d0b490c1da7834c1da7834c1da783400000000 07000000c022aac5c1da78b8c022b696c022c76200000046c1d0b534c1d0b424c1d0b490Call Trace: [<c022aac5>] __generic_unplug_device+0x25/0x30 [<c022b696>] generic_unplug_device+0x6/0x10 [<c022c762>] blk_unplug_work+0x42/0xa0 [<c022c720>] blk_unplug_work+0x0/0xa0 [<c012a307>] run_workqueue+0x97/0x110 [<c012aa00>] worker_thread+0x0/0xe0 [<c012aa7d>] worker_thread+0x7d/0xe0 [<c012d5e0>] autoremove_wake_function+0x0/0x40 [<c012aa00>] worker_thread+0x0/0xe0 [<c012d292>] kthread+0x42/0x70 [<c012d250>] kthread+0x0/0x70 [<c01041df>] kernel_thread_helper+0x7/0x18 ======================= Code: Bad EIP value.
Quoted 2 lines
EIP: [<14001000>]
0x14001000SS:ESP 0068:dfe21f08
So it looks like q->request_fn points at 0x14001000, which is in outer space.
I wonder how that could happen, in the middle of heavy IO operations. Possibly a memory scribble. I'd suggest you enable CONFIG_SLAB, CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC. You could also try switching from CONFIG_SLAB to CONFIG_SLUB, then enable CONFIG_SLUB_DEBUG.
Hi Andrew, hi everybody!
Thanks for the reply!
Quoted 6 lines · Andrew Morton
Quoted 5 lines
The oopses:
#v+
BUG: unable to handle kernel paging request at virtual address 14001000[...]
Quoted 6 lines
So it looks like
q->request_fnpoints at0x14001000, which is in outer space.I wonder how that could happen, in the middle of heavy IO operations. Possibly a memory scribble. I'd suggest you enable
CONFIG_SLAB,CONFIG_DEBUG_SLAB,CONFIG_DEBUG_PAGEALLOC.
That doesn't change anything, apart from adding a line "DEBUG_PAGEALLOC" to the oopses. Do I have to enable the debugging manually? After a quick grep over the sources and a google-session I think not but I'm no kernel-hacker.
Quoted 2 lines
You could also try switching
fromCONFIG_SLABtoCONFIG_SLUB, then enableCONFIG_SLUB_DEBUG.
Unfortunately the only change seems to be, that the Keyboard-Leds don't flash after the crash. Everything else is the same.
Could a hardware-error cause this?
Thanks again,
Signature · 1 line
Philipp Tölke
Quoted 14 lines · On Fri, 03 Aug 2007 22:46:14 +0200 Philipp T__lke <[email protected]> wrote:
Hi Andrew, hi everybody!
Thanks for the reply!
Quoted 9 lines · Andrew Morton
Quoted 5 lines
The oopses:
#v+
BUG: unable to handle kernel paging request at virtual address 14001000[...]
So it looks like
q->request_fnpoints at0x14001000, which is in outer space.
Quoted 6 lines
Quoted 3 lines
I wonder how that could happen, in the middle of heavy IO operations. Possibly a memory scribble. I'd suggest you enable
CONFIG_SLAB,CONFIG_DEBUG_SLAB,CONFIG_DEBUG_PAGEALLOC.That doesn't change anything, apart from adding a line "
DEBUG_PAGEALLOC" to the oopses.
drat.
Quoted 3 lines
Do I have to enable the debugging manually? After a quick grep over the sources and a google-session I think not but I'm no kernel-hacker.
No, just enable it.
Quoted 5 lines
Quoted 2 lines
You could also try switching
fromCONFIG_SLABtoCONFIG_SLUB, then enableCONFIG_SLUB_DEBUG.Unfortunately the only change seems to be, that the Keyboard-Leds don't flash after the crash. Everything else is the same.
drat.
Quoted 1 line
Could a hardware-error cause this?
It sounds unlikely. One wouldn't expect it to crash in the same way each time.