Skip to contentSkip to conversation

oops in __bounce_end_io_read under kvm

DISCUSSION Christoph Hellwig 2 participants 6 messages Dec 26, 2008 – Jan 12, 2009

lore
Share & tools
lore
#0 lore raw

I've been gettings oopsens from a a virtio_blk callchain when running xfsqa under kvm on a virtio devices frequently but not easily reproducibly. The VM has 1.2 GB of memory and thus highmem enabled, but I don't really understand why we bounce buffer in the guest.

The callstack looks like this:

[  898.476526] BUG: unable to handle kernel paging request at fffba200
[  898.480159] IP: [<c01a7c24>] __bounce_end_io_read+0xd4/0x130
[  898.480159] *pde = 00008067 *pte = 00000000 
[  898.480159] Oops: 0002 [#1] SMP 
[  898.480159] last sysfs file: /sys/class/net/lo/operstate
[  898.480159] Modules linked in:
[  898.480159] 
[  898.480159] Pid: 2062, comm: xfs_repair Not tainted (2.6.28-rc9-xfs #23) 
[  898.480159] EIP: 0060:[<c01a7c24>] EFLAGS: 00010086 CPU: 0
[  898.480159] EIP is at __bounce_end_io_read+0xd4/0x130
[  898.480159] EAX: fffba000 EBX: 00000000 ECX: 00000380 EDX: 00000e00
[  898.480159] ESI: ea5fb200 EDI: fffba200 EBP: f4f8fda0 ESP: f4f8fd74
[  898.480159]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[  898.480159] Process xfs_repair (pid: 2062, ti=f4f8e000 task=f6ad0070 task.ti=f4f8e000)
[  898.480159] Stack:
[  898.480159]  00000380 00000000 f6bf3718 f706a8d8 f706ae20 00000000 f7066578 00000086
[  898.480159]  c01a7ca0 00000000 00000000 f4f8fda8 c01a7cb0 f4f8fdb4 c01dad1d f706a8d8
[  898.480159]  f4f8fdd8 c05117e9 f4f8fdd8 00000046 00000001 00000046 f706a8d8 00001000
[  898.480159] Call Trace:
[  898.480159]  [<c01a7ca0>] ? bounce_end_io_read+0x0/0x20
[  898.480159]  [<c01a7cb0>] ? bounce_end_io_read+0x10/0x20
[  898.480159]  [<c01dad1d>] ? bio_endio+0x1d/0x40
[  898.480159]  [<c05117e9>] ? req_bio_endio+0x89/0xe0
[  898.480159]  [<c05118b2>] ? __end_that_request_first+0x72/0x2b0
[  898.480159]  [<c05119d5>] ? __end_that_request_first+0x195/0x2b0
[  898.480159]  [<c0511caf>] ? __blk_end_request+0x1f/0x50
[  898.480159]  [<c059dffd>] ? blk_done+0x4d/0x90
[  898.480159]  [<c063e11a>] ? vring_interrupt+0x2a/0x40
[  898.480159]  [<c063e622>] ? vp_interrupt+0x72/0xb0
[  898.480159]  [<c0184999>] ? handle_IRQ_event+0x29/0x60
[  898.480159]  [<c01862d2>] ? handle_fasteoi_irq+0x62/0xd0
[  898.480159]  [<c012117e>] ? do_IRQ+0x8e/0xc0
[  898.480159]  [<c011fb74>] ? common_interrupt+0x28/0x30
[  898.480159]  [<c052007b>] ? kobject_move+0x10b/0x120
[  898.480159]  [<c013932f>] ? paravirt_get_lazy_mode+0xf/0x20
[  898.480159]  [<c01385b3>] ? kvm_leave_lazy_mmu+0x63/0x80
[  898.480159]  [<c01a3451>] ? mprotect_fixup+0x2b1/0x400
[  898.480159]  [<c01a36ec>] ? sys_mprotect+0x14c/0x1d0
[  898.480159]  [<c011effe>] ? syscall_call+0x7/0xb
[  898.480159]  [<c0110000>] ? kvm_mmu_pte_write+0x270/0x970
[  898.480159] Code: 4e fc ff 8b 55 ec 8b 04 1a 31 d2 e8 d7 6a f9 ff 8b

4d ec 8b 54 19 04 89 c7 89 d1 c1 e9 02 89 4d d4 8b 4d ec 03 7c 19 08 8b 4d d4 <f3> a5 89 d1 83 e1 03 74 02 f3 a4 31 d2 e8 fa 68 f9 ff f7 45 f0

#1 lore raw
Quoted 50 lines · Christoph Hellwig

I've been gettings oopsens from a a virtio_blk callchain when running xfsqa under kvm on a virtio devices frequently but not easily reproducibly. The VM has 1.2 GB of memory and thus highmem enabled, but I don't really understand why we bounce buffer in the guest.

The callstack looks like this:

[  898.476526] BUG: unable to handle kernel paging request at fffba200
[  898.480159] IP: [<c01a7c24>] __bounce_end_io_read+0xd4/0x130
[  898.480159] *pde = 00008067 *pte = 00000000 
[  898.480159] Oops: 0002 [#1] SMP 
[  898.480159] last sysfs file: /sys/class/net/lo/operstate
[  898.480159] Modules linked in:
[  898.480159] 
[  898.480159] Pid: 2062, comm: xfs_repair Not tainted (2.6.28-rc9-xfs #23) 
[  898.480159] EIP: 0060:[<c01a7c24>] EFLAGS: 00010086 CPU: 0
[  898.480159] EIP is at __bounce_end_io_read+0xd4/0x130
[  898.480159] EAX: fffba000 EBX: 00000000 ECX: 00000380 EDX: 00000e00
[  898.480159] ESI: ea5fb200 EDI: fffba200 EBP: f4f8fda0 ESP: f4f8fd74
[  898.480159]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[  898.480159] Process xfs_repair (pid: 2062, ti=f4f8e000 task=f6ad0070 task.ti=f4f8e000)
[  898.480159] Stack:
[  898.480159]  00000380 00000000 f6bf3718 f706a8d8 f706ae20 00000000 f7066578 00000086
[  898.480159]  c01a7ca0 00000000 00000000 f4f8fda8 c01a7cb0 f4f8fdb4 c01dad1d f706a8d8
[  898.480159]  f4f8fdd8 c05117e9 f4f8fdd8 00000046 00000001 00000046 f706a8d8 00001000
[  898.480159] Call Trace:
[  898.480159]  [<c01a7ca0>] ? bounce_end_io_read+0x0/0x20
[  898.480159]  [<c01a7cb0>] ? bounce_end_io_read+0x10/0x20
[  898.480159]  [<c01dad1d>] ? bio_endio+0x1d/0x40
[  898.480159]  [<c05117e9>] ? req_bio_endio+0x89/0xe0
[  898.480159]  [<c05118b2>] ? __end_that_request_first+0x72/0x2b0
[  898.480159]  [<c05119d5>] ? __end_that_request_first+0x195/0x2b0
[  898.480159]  [<c0511caf>] ? __blk_end_request+0x1f/0x50
[  898.480159]  [<c059dffd>] ? blk_done+0x4d/0x90
[  898.480159]  [<c063e11a>] ? vring_interrupt+0x2a/0x40
[  898.480159]  [<c063e622>] ? vp_interrupt+0x72/0xb0
[  898.480159]  [<c0184999>] ? handle_IRQ_event+0x29/0x60
[  898.480159]  [<c01862d2>] ? handle_fasteoi_irq+0x62/0xd0
[  898.480159]  [<c012117e>] ? do_IRQ+0x8e/0xc0
[  898.480159]  [<c011fb74>] ? common_interrupt+0x28/0x30
[  898.480159]  [<c052007b>] ? kobject_move+0x10b/0x120
[  898.480159]  [<c013932f>] ? paravirt_get_lazy_mode+0xf/0x20
[  898.480159]  [<c01385b3>] ? kvm_leave_lazy_mmu+0x63/0x80
[  898.480159]  [<c01a3451>] ? mprotect_fixup+0x2b1/0x400
[  898.480159]  [<c01a36ec>] ? sys_mprotect+0x14c/0x1d0
[  898.480159]  [<c011effe>] ? syscall_call+0x7/0xb
[  898.480159]  [<c0110000>] ? kvm_mmu_pte_write+0x270/0x970
[  898.480159] Code: 4e fc ff 8b 55 ec 8b 04 1a 31 d2 e8 d7 6a f9 ff 8b
Quoted 2 lines

4d ec 8b 54 19 04 89 c7 89 d1 c1 e9 02 89 4d d4 8b 4d ec 03 7c 19 08 8b 4d d4 <f3> a5 89 d1 83 e1 03 74 02 f3 a4 31 d2 e8 fa 68 f9 ff f7 45 f0

Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 2 lines · Jens Axboe

Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?

Just -rc9 and xfs updates. The only change outside of fs/xfs/ is a new define in fs.h and a couple new helpers in inode.c

#3 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe

Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?

Just -rc9 and xfs updates. The only change outside of fs/xfs/ is a new define in fs.h and a couple new helpers in inode.c

OK, then it's surely a new bug. Could you do a gdb l *0xbla on that address to check where it's faulting?

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe

Is this -rc9 stock + xfs patches, or does it include -mm or -next patches (or the block patches)?

Just -rc9 and xfs updates. The only change outside of fs/xfs/ is a new define in fs.h and a couple new helpers in inode.c

OK, then it's surely a new bug. Could you do a gdb l *0xbla on that address to check where it's faulting?

It took a while to reproduce it again, but I got it again today.

(gdb) l *0xc01a5444
0xc01a5444 is in __bounce_end_io_read (/home/hch/work/xfs/arch/x86/include/asm/string_32.h:35).
30 extern size_t strlen(const char *s);
31
32 static __always_inline void *__memcpy(void *to, const void *from, size_t n)
33 {
34 int d0, d1, d2;
35 asm volatile("rep ; movsl\n\t"

36			     "movl %4,%%ecx\n\t"
37			     "andl $3,%%ecx\n\t"
38			     "jz 1f\n\t"
39			     "rep ; movsb\n\t"