oops in __bounce_end_io_read under kvm
I've been gettings oopsens from a a virtio_blk callchain when running xfsqa under kvm on a virtio devices frequently but not easily reproducibly. The VM has 1.2 GB of memory and thus highmem enabled, but I don't really understand why we bounce buffer in the guest.
The callstack looks like this:
[ 898.476526] BUG: unable to handle kernel paging request at fffba200 [ 898.480159] IP: [<c01a7c24>] __bounce_end_io_read+0xd4/0x130 [ 898.480159] *pde = 00008067 *pte = 00000000 [ 898.480159] Oops: 0002 [#1] SMP [ 898.480159] last sysfs file: /sys/class/net/lo/operstate [ 898.480159] Modules linked in: [ 898.480159] [ 898.480159] Pid: 2062, comm: xfs_repair Not tainted (2.6.28-rc9-xfs #23) [ 898.480159] EIP: 0060:[<c01a7c24>] EFLAGS: 00010086 CPU: 0 [ 898.480159] EIP is at __bounce_end_io_read+0xd4/0x130 [ 898.480159] EAX: fffba000 EBX: 00000000 ECX: 00000380 EDX: 00000e00 [ 898.480159] ESI: ea5fb200 EDI: fffba200 EBP: f4f8fda0 ESP: f4f8fd74 [ 898.480159] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 898.480159] Process xfs_repair (pid: 2062, ti=f4f8e000 task=f6ad0070 task.ti=f4f8e000) [ 898.480159] Stack: [ 898.480159] 00000380 00000000 f6bf3718 f706a8d8 f706ae20 00000000 f7066578 00000086 [ 898.480159] c01a7ca0 00000000 00000000 f4f8fda8 c01a7cb0 f4f8fdb4 c01dad1d f706a8d8 [ 898.480159] f4f8fdd8 c05117e9 f4f8fdd8 00000046 00000001 00000046 f706a8d8 00001000 [ 898.480159] Call Trace: [ 898.480159] [<c01a7ca0>] ? bounce_end_io_read+0x0/0x20 [ 898.480159] [<c01a7cb0>] ? bounce_end_io_read+0x10/0x20 [ 898.480159] [<c01dad1d>] ? bio_endio+0x1d/0x40 [ 898.480159] [<c05117e9>] ? req_bio_endio+0x89/0xe0 [ 898.480159] [<c05118b2>] ? __end_that_request_first+0x72/0x2b0 [ 898.480159] [<c05119d5>] ? __end_that_request_first+0x195/0x2b0 [ 898.480159] [<c0511caf>] ? __blk_end_request+0x1f/0x50 [ 898.480159] [<c059dffd>] ? blk_done+0x4d/0x90 [ 898.480159] [<c063e11a>] ? vring_interrupt+0x2a/0x40 [ 898.480159] [<c063e622>] ? vp_interrupt+0x72/0xb0 [ 898.480159] [<c0184999>] ? handle_IRQ_event+0x29/0x60 [ 898.480159] [<c01862d2>] ? handle_fasteoi_irq+0x62/0xd0 [ 898.480159] [<c012117e>] ? do_IRQ+0x8e/0xc0 [ 898.480159] [<c011fb74>] ? common_interrupt+0x28/0x30 [ 898.480159] [<c052007b>] ? kobject_move+0x10b/0x120 [ 898.480159] [<c013932f>] ? paravirt_get_lazy_mode+0xf/0x20 [ 898.480159] [<c01385b3>] ? kvm_leave_lazy_mmu+0x63/0x80 [ 898.480159] [<c01a3451>] ? mprotect_fixup+0x2b1/0x400 [ 898.480159] [<c01a36ec>] ? sys_mprotect+0x14c/0x1d0 [ 898.480159] [<c011effe>] ? syscall_call+0x7/0xb [ 898.480159] [<c0110000>] ? kvm_mmu_pte_write+0x270/0x970 [ 898.480159] Code: 4e fc ff 8b 55 ec 8b 04 1a 31 d2 e8 d7 6a f9 ff 8b
4d ec 8b 54 19 04 89 c7 89 d1 c1 e9 02 89 4d d4 8b 4d ec 03 7c 19 08 8b 4d d4 <f3> a5 89 d1 83 e1 03 74 02 f3 a4 31 d2 e8 fa 68 f9 ff f7 45 f0
Quoted 50 lines · Christoph Hellwig
I've been gettings oopsens from a a
virtio_blkcallchain when running xfsqa under kvm on a virtio devices frequently but not easily reproducibly. The VM has 1.2 GB of memory and thus highmem enabled, but I don't really understand why we bounce buffer in the guest.The callstack looks like this:
[ 898.476526] BUG: unable to handle kernel paging request at fffba200 [ 898.480159] IP: [<c01a7c24>] __bounce_end_io_read+0xd4/0x130 [ 898.480159] *pde = 00008067 *pte = 00000000 [ 898.480159] Oops: 0002 [#1] SMP [ 898.480159] last sysfs file: /sys/class/net/lo/operstate [ 898.480159] Modules linked in: [ 898.480159] [ 898.480159] Pid: 2062, comm: xfs_repair Not tainted (2.6.28-rc9-xfs #23) [ 898.480159] EIP: 0060:[<c01a7c24>] EFLAGS: 00010086 CPU: 0 [ 898.480159] EIP is at __bounce_end_io_read+0xd4/0x130 [ 898.480159] EAX: fffba000 EBX: 00000000 ECX: 00000380 EDX: 00000e00 [ 898.480159] ESI: ea5fb200 EDI: fffba200 EBP: f4f8fda0 ESP: f4f8fd74 [ 898.480159] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 898.480159] Process xfs_repair (pid: 2062, ti=f4f8e000 task=f6ad0070 task.ti=f4f8e000) [ 898.480159] Stack: [ 898.480159] 00000380 00000000 f6bf3718 f706a8d8 f706ae20 00000000 f7066578 00000086 [ 898.480159] c01a7ca0 00000000 00000000 f4f8fda8 c01a7cb0 f4f8fdb4 c01dad1d f706a8d8 [ 898.480159] f4f8fdd8 c05117e9 f4f8fdd8 00000046 00000001 00000046 f706a8d8 00001000 [ 898.480159] Call Trace: [ 898.480159] [<c01a7ca0>] ? bounce_end_io_read+0x0/0x20 [ 898.480159] [<c01a7cb0>] ? bounce_end_io_read+0x10/0x20 [ 898.480159] [<c01dad1d>] ? bio_endio+0x1d/0x40 [ 898.480159] [<c05117e9>] ? req_bio_endio+0x89/0xe0 [ 898.480159] [<c05118b2>] ? __end_that_request_first+0x72/0x2b0 [ 898.480159] [<c05119d5>] ? __end_that_request_first+0x195/0x2b0 [ 898.480159] [<c0511caf>] ? __blk_end_request+0x1f/0x50 [ 898.480159] [<c059dffd>] ? blk_done+0x4d/0x90 [ 898.480159] [<c063e11a>] ? vring_interrupt+0x2a/0x40 [ 898.480159] [<c063e622>] ? vp_interrupt+0x72/0xb0 [ 898.480159] [<c0184999>] ? handle_IRQ_event+0x29/0x60 [ 898.480159] [<c01862d2>] ? handle_fasteoi_irq+0x62/0xd0 [ 898.480159] [<c012117e>] ? do_IRQ+0x8e/0xc0 [ 898.480159] [<c011fb74>] ? common_interrupt+0x28/0x30 [ 898.480159] [<c052007b>] ? kobject_move+0x10b/0x120 [ 898.480159] [<c013932f>] ? paravirt_get_lazy_mode+0xf/0x20 [ 898.480159] [<c01385b3>] ? kvm_leave_lazy_mmu+0x63/0x80 [ 898.480159] [<c01a3451>] ? mprotect_fixup+0x2b1/0x400 [ 898.480159] [<c01a36ec>] ? sys_mprotect+0x14c/0x1d0 [ 898.480159] [<c011effe>] ? syscall_call+0x7/0xb [ 898.480159] [<c0110000>] ? kvm_mmu_pte_write+0x270/0x970 [ 898.480159] Code: 4e fc ff 8b 55 ec 8b 04 1a 31 d2 e8 d7 6a f9 ff 8b
Quoted 2 lines
4d ec 8b 54 19 04 89 c7 89 d1 c1 e9 02 89 4d d4 8b 4d ec 03 7c 19 08 8b 4d d4 <f3> a5 89 d1 83 e1 03 74 02 f3 a4 31 d2 e8 fa 68 f9 ff f7 45 f0
Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?
Signature · 1 line
Jens Axboe
Quoted 2 lines · Jens Axboe
Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?
Just -rc9 and xfs updates. The only change outside of fs/xfs/ is a new define in fs.h and a couple new helpers in inode.c
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe
Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?
Just -rc9 and xfs updates. The only change outside of
fs/xfs/is a new define infs.hand a couple new helpers ininode.c
OK, then it's surely a new bug. Could you do a gdb l *0xbla on that address to check where it's faulting?
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe
Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?
Just -rc9 and xfs updates. The only change outside of
fs/xfs/is a new define infs.hand a couple new helpers ininode.cOK, then it's surely a new bug. Could you do a gdb l *0xbla on that address to check where it's faulting?
It took a while to reproduce it again, but I got it again today.
(gdb) l *0xc01a54440xc01a5444 is in __bounce_end_io_read (/home/hch/work/xfs/arch/x86/include/asm/string_32.h:35).
30 extern size_t strlen(const char *s);
31
32 static __always_inline void *__memcpy(void *to, const void *from, size_t n)
33 {
34 int d0, d1, d2;
35 asm volatile("rep ; movsl\n\t"
36 "movl %4,%%ecx\n\t" 37 "andl $3,%%ecx\n\t" 38 "jz 1f\n\t" 39 "rep ; movsb\n\t"
FYI this is so far 100% reproducible when running xfsqa twice without rebooting the virtual machine.