Oops in bdflush with 2.4.1[4|7]-xfs

Quoted 1 line

EIP; c0192fe9 <pagebuf_lookup+79/8c> <=====

We've got a couple of development machines with hardware raid sets that we use XFS on. A few weeks ago, while running 2.4.14-xfs, there was an Oops in bdflush, which caused bdflush to go defunct, and kupdated to go into some sort of loop, chewing one of the CPUs. I upgraded the machine to 2.4.17-xfs, and for a few weeks it's been fine, but then this afternoon there was again an oops in bdflush, which again caused bdflush to go defunct, and kupdated to chew a CPU (along with an attempted sync, the shutdown process, etc). Below is the decoded oops. Any help is appreciated. Thanks.

ksymoops 0.7c on i686 2.4.17-xfs. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.17-xfs/ (default)
-m /boot/System.map-2.4.17-xfs (specified)

Unable to handle kernel NULL pointer dereference at virtual address
00000030
c0192fe9
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0192fe9>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 00000000
esi: 00000000 edi: 00000001 ebp: 00000000 esp: c4c49864
ds: 0018 es: 0018 ss: 0018
Process bdflush (pid: 6, stackpage=c4c49000)

Stack: e5187ed8 00000000 00000000 c4c4996c c4c4996c c4c49b68 c4c49b68
0b000003 
       f3242be0 00000000 00000004 cbf27200 c4c498c8 f3242be0 00000046
0b000010 
       00000000 00000001 f79fd8f4 00000000 00000016 f71a5400 00000000
00000002 
Call Trace: [<c0194b68>] [<c018f9ad>] [<c018f746>] [<c01919ce>]

[<c01a103e>]
[<c021a41e>] [<c02208d2>] [<c022004a>] [<c0220244>] [<c02204b1>]
[<c0238573>]
[<c01ad063>] [<c01ad063>] [<c01a34b3>] [<c01a515f>] [<c0238573>]
[<c01d12fb>]
[<c01d23a6>] [<c01efd74>] [<c012b965>] [<c012b9ac>] [<c012c270>]
[<c01edcf3>]
[<c0182854>] [<c018188b>] [<c01edc78>] [<c01ede8b>] [<c01edc78>]
[<c0133745>]
[<c0133909>] [<c01369a4>] [<c0105000>] [<c01055db>]

Code: 8b 52 30 89 54 24 58 51 55 8b 44 24 60 50 8b 54 24 78 52 e8 

Quoted 1 line

Quoted 1 line

EIP; c0192fe9 <pagebuf_lookup+79/8c> <=====

Trace; c0194b68 <_pagebuf_handle_iovecs+dc/f0> Trace; c018f9ad <nlm4svc_callback+19/98> Trace; c018f746 <nlm4svc_proc_unshare+2/bc> Trace; c01919ce <devpts_root_readdir+12e/134> Trace; c01a103e <xfs_dm_punch_hole+1e/27c> Trace; c021a41e <vt_ioctl+6ce/2050>
Trace; c02208d2 <do_con_trol+996/e28> Trace; c022004a <do_con_trol+10e/e28> Trace; c0220244 <do_con_trol+308/e28> Trace; c02204b1 <do_con_trol+575/e28> Trace; c0238573 <scsi_init_cmd_errh+a7/e0> Trace; c01ad063 <xfs_attr_leaf_toosmall+1af/244> Trace; c01ad063 <xfs_attr_leaf_toosmall+1af/244> Trace; c01a34b3 <xfs_alloc_ag_vextent_exact+9b/178> Trace; c01a515f <xfs_alloc_read_agf+93/23c> Trace; c0238573 <scsi_init_cmd_errh+a7/e0> Trace; c01d12fb <xfs_dir2_block_to_sf+f3/2f8> Trace; c01d23a6 <xfs_dir2_sf_lookup+c6/280> Trace; c01efd74 <xfs_trans_init+1384/1f30> Trace; c012b965 <vmalloc_area_pages+35/1b0> Trace; c012b9ac <vmalloc_area_pages+7c/1b0> Trace; c012c270 <kmem_cache_destroy+7c/fc> Trace; c01edcf3 <xfs_rename_target_checks+33/b8> Trace; c0182854 <nfsd_cache_init+124/138> Trace; c018188b <exp_rootfh+10f/278>
Trace; c01edc78 <xfs_rename_error_checks+c0/108> Trace; c01ede8b <xfs_rename+113/c78>
Trace; c01edc78 <xfs_rename_error_checks+c0/108> Trace; c0133745 <sys_ftruncate+4d/154> Trace; c0133909 <sys_truncate64+bd/184> Trace; c01369a4 <fsync_inode_buffers+140/18c> Trace; c0105000 <_stext+0/0>
Trace; c01055db <kernel_thread+13/30>
Code; c0192fe9 <pagebuf_lookup+79/8c> 00000000 <_EIP>:
Code; c0192fe9 <pagebuf_lookup+79/8c> <=====
0: 8b 52 30 mov 0x30(%edx),%edx <=====
Code; c0192fec <pagebuf_lookup+7c/8c>
3: 89 54 24 58 mov %edx,0x58(%esp,1)
Code; c0192ff0 <pagebuf_lookup+80/8c>
7: 51 push %ecx
Code; c0192ff1 <pagebuf_lookup+81/8c>
8: 55 push %ebp
Code; c0192ff2 <pagebuf_lookup+82/8c>
9: 8b 44 24 60 mov 0x60(%esp,1),%eax
Code; c0192ff6 <pagebuf_lookup+86/8c>
d: 50 push %eax
Code; c0192ff7 <pagebuf_lookup+87/8c>
e: 8b 54 24 78 mov 0x78(%esp,1),%edx
Code; c0192ffb <pagebuf_lookup+8b/8c>
12: 52 push %edx
Code; c0192ffc <pagebuf_readahead+0/3c>
13: e8 00 00 00 00 call 18 <_EIP+0x18> c0193001
<pagebuf_readahead+5/3c>

Could you see if my XFS-AA patch does anything for you? There are changes to bdflush in it and I'd be interested to see if they go away.

http://www.digitalroadkill.net/Patches/2.4.17-xfs-aa.patch.bz2

Thus spake Austin Gonyou:

Quoted 5 lines

Could you see if my XFS-AA patch does anything for you? There are changes to bdflush in it and I'd be interested to see if they go away.

http://www.digitalroadkill.net/Patches/2.4.17-xfs-aa.patch.bz2

Unfortunately I can't really do to much messing around with this machine right now, it's being used pretty heavily. Even after bdflush died and I needed to bounce the machine, I just about had to beat the developers off the machine with a stick. :-)

Eric Sandeen suggested turning off DMAPI support, so I'm going to give that a try first. I'll go ahead and grab a copy of your patch, and give it a try if the problem still resurfaces.

Signature · 11 lines

Nathan Poznick <[email protected]>
PGP Key: http://drunkmonkey.org/pgpkey.txt

Boss:   You forgot to assign the result of your map!
Hacker: Dang, I'm always forgetting my assignations...
Boss:   And what's that "goto" doing there?!?
Hacker: Er, I guess my finger slipped when I was typing
"getservbyport"...
Boss:   Ah well, accidents will happen.  Maybe we should have picked
APL.
-- Larry Wall