2.6.13-rc6 Oops with Software RAID, LVM, JFS, NFS
Hi,
I posted an oops a few days ago from 2.6.12.3 [1]. Here are the results of my tests on 2.6.13-rc6. The kernel oopses, but it the box isn't completely hosed; I can still log in and move around. It appears that the only things that are locked are the apps that were doing i/o to the test partition. More detailed info about my configuration can be found here:
<http://www.icglink.com/debug-2.6.13-rc6.html>
Here is the oops:
Oops: 0000 [#1] SMP Modules linked in: CPU: 0 EIP: 0060:[<c0116dd0>] Not tainted VLI EFLAGS: 00010207 (2.6.13-rc6)
EIP is at kmap+0x10/0x30
eax: 00000003 ebx: d0977440 ecx: c9efb470 edx: 00000000
esi: c1000000 edi: 00000000 ebp: ce59d570 esp: f7adde18
ds: 007b es: 007b ss: 0068
Process md4_raid1 (pid: 6442, threadinfo=f7adc000 task=f70eda20)
Stack: c014e0bd c9efb470 00000001 00000000 00000000 cb129000 00000001 e0c65f00
f7dcbe18 087641ef 00000000 f7dcbe18 c014e146 f7dcbe18 f7addeb0 c3146940
c033f03f f7dcbe18 f7addeb0 0021d906 00000000 0000003f 00000040 00000000
Call Trace:
[<c014e0bd>] __blk_queue_bounce+0x20d/0x260
[<c014e146>] blk_queue_bounce+0x36/0x60
[<c033f03f>] __make_request+0x5f/0x560
[<c0132b90>] autoremove_wake_function+0x0/0x60
[<c033f921>] generic_make_request+0x151/0x230
[<c0132b90>] autoremove_wake_function+0x0/0x60
[<c04a5fac>] schedule+0x62c/0xcb0
[<c04a5fe0>] schedule+0x660/0xcb0
[<c0132b90>] autoremove_wake_function+0x0/0x60
[<c0126973>] del_timer+0x73/0x80
[<c033db0d>] blk_remove_plug+0x3d/0x80
[<c03feed9>] raid1d+0x289/0x2a0
[<c0414bb3>] md_thread+0x143/0x190
[<c0132b90>] autoremove_wake_function+0x0/0x60
[<c0102ed2>] ret_from_fork+0x6/0x14
[<c0132b90>] autoremove_wake_function+0x0/0x60
[<c0414a70>] md_thread+0x0/0x190
[<c01011b5>] kernel_thread_helper+0x5/0x10
Code: 00 40 c7 46 0c 90 30 15 c0 c7 46 10 90 31 15 c0 eb b9 90 90 90 90 90 90 90 90 90 8b 4c 24 04 8b 01 c1 e8 1e 8b 14 85 14 f4 63 c0 <8b> 82 0c 04 00 00 05 00 09 00 00 39 c2 74 05 e9 ac 73 03 00 89Thanks for looking..
Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733) /* vim:set noai nocindent ts=8 sw=8: */
Quoted 9 lines · On Thursday August 11, [email protected] wrote:
Hi,
I posted an oops a few days ago from 2.6.12.3 [1]. Here are the results of my tests on 2.6.13-rc6. The kernel oopses, but it the box isn't completely hosed; I can still log in and move around. It appears that the only things that are locked are the apps that were doing i/o to the test partition. More detailed info about my configuration can be found here:
You don't seem to give details on how lvm is used to combine the md arrays, though I'm not sure that would help particularly.
Quoted 14 lines
Here is the oops:
Oops: 0000 [#1] SMP Modules linked in: CPU: 0 EIP: 0060:[<c0116dd0>] Not tainted VLI EFLAGS: 00010207 (2.6.13-rc6)EIP is at
kmap+0x10/0x30
eax: 00000003 ebx:d0977440ecx:c9efb470edx: 00000000
esi:c1000000edi: 00000000 ebp:ce59d570esp:f7adde18
ds: 007b es: 007b ss: 0068
Processmd4_raid1(pid: 6442, threadinfo=f7adc000 task=f70eda20)
Quoted 5 lines
Stack: c014e0bd c9efb470 00000001 00000000 00000000 cb129000 00000001 e0c65f00
f7dcbe18 087641ef 00000000 f7dcbe18 c014e146 f7dcbe18 f7addeb0 c3146940
c033f03f f7dcbe18 f7addeb0 0021d906 00000000 0000003f 00000040 00000000
Call Trace:
[<c014e0bd>] __blk_queue_bounce+0x20d/0x260--snip---
Quoted 2 lines
Code: 00 40 c7 46 0c 90 30 15 c0 c7 46 10 90 31 15 c0 eb b9 90 90 90 90 90 90 90 90 90 8b 4c 24 04 8b 01 c1 e8 1e 8b 14 85 14 f4 63 c0 <8b> 82 0c 04 00 00 05 00 09 00 00 39 c2 74 05 e9 ac 73 03 00 89
The code is Oopsing in a call to kmap in arch/i386/highmem.c The PageHighMem macro calls is_highmem(page_zone(page)). page_zone is defined in mm.h
static inline struct zone *page_zone(struct page *page)
{
return zone_table[(page->flags >> ZONETABLE_PGSHIFT) &
ZONETABLE_MASK];
}Now at the point of the crash, eax is
(page->flags >> ZONETABLE_PGSHIFT),
which is '3'. So it seems that this page is in zone 3. However zone_table[3] is now in edx, and we can see it is '0'. There are only 3 zones (normal, dma, highmem), so nothing should ever by in zone 3. This page is clearly bad.
However that is as far as I can get. I don't know whether this is a bad page pointer passed down from jfs or nfsd, a page pointer that was corrupted by either lvm or md, or a valid page pointer that has managed to get a bad zone number encoded in it's flags.
You could possibly put something like
struct bio_vec *from; int i; bio_for_each_segment(from, bio, i) BUG_ON(page_zone(from->bv_page)==NULL);
in generic_make_requst in drivers/block/ll_rw_blk.c, just before the call to q->make_request_fn.
This might trigger the bug early enough to see what is happening.
Quoted 3 lines
Thanks for looking..
Thanks for testing.
NeilBrown
Quoted 9 lines · On Fri, 12 Aug 2005 12:07:21 +1000 Neil Brown <[email protected]> wrote:
Quoted 8 lines · On Thursday August 11, [email protected] wrote:
Hi,
I posted an oops a few days ago from 2.6.12.3 [1]. Here are the results of my tests on 2.6.13-rc6. The kernel oopses, but it the box isn't completely hosed; I can still log in and move around. It appears that the only things that are locked are the apps that were doing i/o to the test partition. More detailed info about my configuration can be found here:
Quoted 5 lines
Quoted 1 line
You don't seem to give details on how lvm is used to combine the md arrays, though I'm not sure that would help particularly.
FYI:
vgdisplay -v vg1
Using volume group(s) on command line
Finding volume group "vg1"
--- Volume group ---
VG Name vg1
System ID
Format lvm2 Metadata Areas 2 Metadata Sequence No 8 VG Access read/write VG Status resizable MAX LV 255 Cur LV 1 Open LV 0 Max PV 255 Cur PV 2 Act PV 2 VG Size 410.00 GB PE Size 128.00 MB Total PE 3280 Alloc PE / Size 1093 / 136.62 GB Free PE / Size 2187 / 273.38 GB VG UUID XuRomW-O6Uw-oQGq-vdwD-YwMT-Dltj-NExFmV
--- Logical volume ---
LV Name /dev/vg1/home VG Name vg1 LV UUID K7Gq9l-Vjte-ksFt-s0vn-ejqT-RGYc-5Aibtx LV Write Access read/write LV Status available # open 0 LV Size 136.62 GB Current LE 1093 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:3
--- Physical volumes ---
PV Name /dev/md4 PV UUID VgHU6k-lZmE-j686-dvfX-OSsM-yh28-Jyfidn PV Status allocatable Total PE / Free PE 1093 / 0
PV Name /dev/md7 PV UUID n4rVmy-rARO-a5mY-Iiqo-GvOx-2nbG-HluaTa PV Status allocatable Total PE / Free PE 2187 / 2187
md7 is in there to test live migration from smaller disks to larger ones.
Quoted 9 lines
struct bio_vec *from; int i; bio_for_each_segment(from, bio, i) BUG_ON(page_zone(from->bv_page)==NULL);in
generic_make_requstindrivers/block/ll_rw_blk.c, just before the call toq->make_request_fn.
This might trigger the bug early enough to see what is happening.
I'll try this and report the results.
Signature · 3 lines
Phil Dier <[email protected]> /* vim:set ts=8 sw=8 nocindent noai: */
Quoted 10 lines · On Fri, 12 Aug 2005 12:07:21 +1000 Neil Brown <[email protected]> wrote:
You could possibly put something like
struct bio_vec *from; int i; bio_for_each_segment(from, bio, i) BUG_ON(page_zone(from->bv_page)==NULL);in
generic_make_requstindrivers/block/ll_rw_blk.c, just before the call toq->make_request_fn.
This might trigger the bug early enough to see what is happening.
I've got tests running with this code in place, by I/O is so slow now I don't think it's going to oops (or if it does, it'll be a while)..
Is there any other info I can collect to help track this down?
Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733) /* vim:set noai nocindent ts=8 sw=8: */
Quoted 14 lines · Phil Dier
Quoted 10 lines · On Fri, 12 Aug 2005 12:07:21 +1000 Neil Brown <[email protected]> wrote:
You could possibly put something like
struct bio_vec *from; int i; bio_for_each_segment(from, bio, i) BUG_ON(page_zone(from->bv_page)==NULL);in
generic_make_requstindrivers/block/ll_rw_blk.c, just before the call toq->make_request_fn.
This might trigger the bug early enough to see what is happening.
Quoted 4 lines
I've got tests running with this code in place, by I/O is so slow now I don't think it's going to oops (or if it does, it'll be a while)..
Is there any other info I can collect to help track this down?
Well, while we are slowing things down in the name of debugging.. you might try setting the following debug options in your config:
CONFIG_DEBUG_PAGEALLOCCONFIG_DEBUG_HIGHMEMCONFIG_DEBUG_SLABCONFIG_FRAME_POINTER
Can anyone think of anything else?
According to the website you don't have these on right now.
Sonny
I just got this:
Unable to handle kernel paging request at virtual address eeafefc0 printing eip: c0188487 *pde = 00681067 *pte = 2eafe000 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: CPU: 1 EIP: 0060:[<c0188487>] Not tainted VLI EFLAGS: 00010296 (2.6.13-rc6)
EIP is at inotify_inode_queue_event+0x17/0x130
eax: eeafefc0 ebx: 00000000 ecx: 00000200 edx: eeafee9c
esi: 00000000 edi: ef4cbe9c ebp: f66e1eac esp: f66e1e84
ds: 007b es: 007b ss: 0068
Process nfsd (pid: 6259, threadinfo=f66e0000 task=f6307b00)
Stack: eeafee9c c0536a34 ee900f6c f66e1eac c0179949 eeafefc0 23644d80 00000000
00000000 ef4cbe9c f66e1ed4 c01713ad eeafee9c 00000400 00000000 00000000
eeafee9c ee900f6c f0940f6c f6f0adf8 f66e1f00 c020caa1 ef4cbe9c ee900f6c
Call Trace:
[<c0103e7f>] show_stack+0x7f/0xa0
[<c0104030>] show_registers+0x160/0x1d0
[<c0104260>] die+0x100/0x180
[<c0116199>] do_page_fault+0x369/0x6ed
[<c0103aa3>] error_code+0x4f/0x54
[<c01713ad>] vfs_unlink+0x17d/0x210
[<c020caa1>] nfsd_unlink+0x161/0x240
[<c0207c64>] nfsd_proc_remove+0x44/0x90
[<c0206747>] nfsd_dispatch+0xd7/0x200
[<c0491b13>] svc_process+0x533/0x670
[<c02064dd>] nfsd+0x1bd/0x350
[<c01011e5>] kernel_thread_helper+0x5/0x10
Code: ff ff ff 8b 5d f8 8b 75 fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 57 56 53 83 ec 1c 8b 45 08 8b 55 08 05 24 01 00 00 89 45 ec <39> 82 24 01 00 00 74 5d f0 ff 8a 2c 01 00 00 0f 88 d1 0b 00 00Signature · 3 lines
Phil Dier <[email protected]> /* vim:set ts=8 sw=8 nocindent noai: */
Quoted 27 lines · Phil Dier
I just got this:
Unable to handle kernel paging request at virtual address eeafefc0 printing eip: c0188487 *pde = 00681067 *pte = 2eafe000 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: CPU: 1 EIP: 0060:[<c0188487>] Not tainted VLI EFLAGS: 00010296 (2.6.13-rc6)EIP is at
inotify_inode_queue_event+0x17/0x130
eax:eeafefc0ebx: 00000000 ecx: 00000200 edx:eeafee9c
esi: 00000000 edi:ef4cbe9cebp:f66e1eacesp:f66e1e84
ds: 007b es: 007b ss: 0068
Process nfsd (pid: 6259, threadinfo=f66e0000 task=f6307b00)Stack: eeafee9c c0536a34 ee900f6c f66e1eac c0179949 eeafefc0 23644d80 00000000 00000000 ef4cbe9c f66e1ed4 c01713ad eeafee9c 00000400 00000000 00000000 eeafee9c ee900f6c f0940f6c f6f0adf8 f66e1f00 c020caa1 ef4cbe9c ee900f6c Call Trace: [<c0103e7f>] show_stack+0x7f/0xa0 [<c0104030>] show_registers+0x160/0x1d0 [<c0104260>] die+0x100/0x180 [<c0116199>] do_page_fault+0x369/0x6ed [<c0103aa3>] error_code+0x4f/0x54
Quoted 8 lines
[<c01713ad>] vfs_unlink+0x17d/0x210 [<c020caa1>] nfsd_unlink+0x161/0x240 [<c0207c64>] nfsd_proc_remove+0x44/0x90 [<c0206747>] nfsd_dispatch+0xd7/0x200 [<c0491b13>] svc_process+0x533/0x670 [<c02064dd>] nfsd+0x1bd/0x350 [<c01011e5>] kernel_thread_helper+0x5/0x10 Code: ff ff ff 8b 5d f8 8b 75 fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 57 56 53 83 ec 1c 8b 45 08 8b 55 08 05 24 01 00 00 89 45 ec <39> 82 24 01 00 00 74 5d f0 ff 8a 2c 01 00 00 0f 88 d1 0b 00 00
int vfs_unlink(struct inode *dir, struct dentry *dentry) {
<snipped>
/* We don't d_delete() NFS sillyrenamed files--they still exist.
*/
if (!error && !(dentry->d_flags & DCACHE_NFSFS_RENAMED)) {
struct inode *inode = dentry->d_inode;
d_delete(dentry);
<==========
fsnotify_unlink(dentry, inode, dir);
}
return error;
}static inline void fsnotify_unlink(struct dentry *dentry, struct inode *inode, struct inode *dir) {
<snipped>
inotify_inode_queue_event(inode, IN_DELETE_SELF, 0, NULL); <=====
<snipped>
}
void inotify_inode_queue_event(struct inode *inode, u32 mask, u32 cookie,
const char *name)
{
struct inotify_watch *watch, *next;
if (!inotify_inode_watched(inode)) <======
return;<snipped>
}
static inline int inotify_inode_watched(struct inode *inode)
{
return !list_empty(&inode->inotify_watches); <===
}I'm new here, if the inode isn't being watched, what's to stop d_delete from removing the inode before fsnotify_unlink proceeds to use it?
Thanks,
Zwane
Quoted 2 lines · Zwane Mwaikambo
I'm new here, if the inode isn't being watched, what's to stop
d_deletefrom removing the inode beforefsnotify_unlinkproceeds to use it?
Nothing. But check out
Should solve this problem?
Robert Love
Quoted 8 lines · Robert Love
Quoted 2 lines · Zwane Mwaikambo
I'm new here, if the inode isn't being watched, what's to stop
d_deletefrom removing the inode beforefsnotify_unlinkproceeds to use it?Nothing. But check out
That git web interface looks rather spiffy.
Quoted 1 line
Should solve this problem?
Seems to fit the bill perfectly.
Thanks,
Zwane
Quoted 11 lines · On Sun, 14 Aug 2005 21:20:35 -0600 (MDT) Zwane Mwaikambo…
Quoted 8 lines · Robert Love
Quoted 2 lines · Zwane Mwaikambo
I'm new here, if the inode isn't being watched, what's to stop
d_deletefrom removing the inode beforefsnotify_unlinkproceeds to use it?Nothing. But check out
Quoted 9 lines
That git web interface looks rather spiffy.
Quoted 1 line
Should solve this problem?
Seems to fit the bill perfectly.
Thanks,
Zwane
So, for the record, I patched my 2.6.13-rc6 kernel with the patch at this location:
and I will be testing it today.
Thanks to all of you guys.
Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733) /* vim:set noai nocindent ts=8 sw=8: */