Crashes and lockups in XFS filesystem (2.6.8-rc4).

Hello, I am getting persistent lockups that could be IMHO XFS-related. I
created a fresh XFS filesystem in a SCSI disk, with xfsprogs version 2.6.18.

Mounted /dev/sda1 under /mnt, after that, I have been copying lots of files
from /dev/md0, then run a find blabla -exec rm \{\{ \; over /mnt and then voilà! the lockup:

ulises:/mnt/debian# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda5             5,6G  1,8G  3,5G  34% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/hda1              19M   11M  6,9M  62% /boot
/dev/hda6             9,2G  1,8G  7,0G  21% /var
/dev/hda8             3,7G  2,3G  1,2G  67% /home
/dev/md0              224G  182G   42G  82% /mirror
/dev/sda1              69G   56G   13G  82% /mnt
ulises:/mnt/debian# find . \( -name *m68k.deb \) -exec rm \{\} \; &
[1] 13215
ulises:/mnt/debian# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda5             5,6G  1,8G  3,5G  34% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/hda1              19M   11M  6,9M  62% /boot
/dev/hda6             9,2G  1,8G  7,0G  21% /var
/dev/hda8             3,7G  2,3G  1,2G  67% /home
Segmentation fault  <<<<<< when trying to display free space in /mnt
ulises:/mnt/debian# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda5              5767428   1839536   3634920  34% /
tmpfs                   257484         0    257484   0% /dev/shm
/dev/hda1                18998     11044      6973  62% /boot
/dev/hda6              9612100   1874984   7248844  21% /var
/dev/hda8              3799944   2398188   1208728  67% /home
Segmentation fault
ulises:~# dmesg
[...]
XFS mounting filesystem sda1
Starting XFS recovery on filesystem: sda1 (dev: sda1)
Ending XFS recovery on filesystem: sda1 (dev: sda1)
Unable to handle kernel paging request at virtual address 020000b4
 printing eip:
c01fcd41
*pde = 00000000
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c01fcd41>]    Not tainted
EFLAGS: 00010206   (2.6.8-rc4)
EIP is at xfs_log_force+0x28/0x6c
eax: 02000000   ebx: 00000002   ecx: 00000000   edx: 00000000
esi: df234c40   edi: dacae2f4   ebp: 00000000   esp: dec44f64
ds: 007b   es: 007b   ss: 0068
Process xfssyncd (pid: 174, threadinfo=dec44000 task=dee7a8c0)
Stack: dffabc80 00000000 00000031 c0207bba 00000031 df234c40 c020f59d dacae2f4
       00000000 00000000 00000002 004bcd64 dec44fb0 00000000 00000002 00000000
       dec44000 df234c40 00000000 00000000 c020ec0f dacae2f4 00000031 00000000
Call Trace:
 [<c0207bba>] xfs_getsb+0x2f/0x45
 [<c020f59d>] xfs_syncsub+0x4e/0x303
 [<c020ec0f>] xfs_sync+0x29/0x2d
 [<c02211cc>] vfs_sync+0x34/0x38
 [<c022076f>] xfssyncd+0x7e/0xce
 [<c02206f1>] xfssyncd+0x0/0xce
 [<c0101fdd>] kernel_thread_helper+0x5/0xb
Code: f6 80 b4 00 00 00 08 75 34 89 ce 09 d6 75 18 89 5c 24 04 89
 <1>Unable to handle kernel NULL pointer dereference at virtual address 
00000000
 printing eip:
c020ebb4
*pde = 00000000
Oops: 0000 [#2]
CPU:    0
EIP:    0060:[<c020ebb4>]    Not tainted
EFLAGS: 00010202   (2.6.8-rc4)
EIP is at xfs_statvfs+0xb6/0xe8
eax: 00000000   ebx: dacae314   ecx: dacae2f4   edx: 00000000
esi: dacae2f4   edi: 00000000   ebp: c0cf1ebc   esp: c0cf1e68
ds: 007b   es: 007b   ss: 0068
Process df (pid: 30308, threadinfo=c0cf1000 task=c64726c0)
Stack: dacae2f4 00000000 c0cf1f74 c0cf1efc c0cf1000 c0221194 dfced400 c0cf1ebc
       00000000 c0220b0a dfced400 c0cf1ebc 00000000 c013e3af c154ce00 c0cf1ebc
       c0cf1f14 c013e430 c154ce00 c0cf1ebc d24a1001 58465342 00000000 bfeb0fdc
Call Trace:
 [<c0221194>] vfs_statvfs+0x34/0x38
 [<c0220b0a>] linvfs_statfs+0x28/0x2e
 [<c013e3af>] vfs_statfs+0x4b/0x66
 [<c013e430>] vfs_statfs64+0x23/0xb2
 [<c013e5ea>] sys_statfs64+0x81/0xbf
 [<c024fac4>] tty_write+0x179/0x1bc
 [<c0254921>] write_chan+0x0/0x219
 [<c024f94b>] tty_write+0x0/0x1bc
 [<c01403e9>] vfs_write+0xc9/0x119
 [<c014050a>] sys_write+0x51/0x80
 [<c0103aa1>] sysenter_past_esp+0x52/0x71
Code: 8b 00 c7 45 24 ff 00 00 00 89 c2 0f b6 c8 25 00 ff 0f 00 c1
 <1>Unable to handle kernel NULL pointer dereference at virtual address 
00000000
 printing eip:
c020ebb4
*pde = 00000000
Oops: 0000 [#3]
CPU:    0
EIP:    0060:[<c020ebb4>]    Not tainted
EFLAGS: 00010202   (2.6.8-rc4)
EIP is at xfs_statvfs+0xb6/0xe8
eax: 00000000   ebx: dacae314   ecx: dacae2f4   edx: 00000000
esi: dacae2f4   edi: 00000000   ebp: c987cebc   esp: c987ce68
ds: 007b   es: 007b   ss: 0068
Process df (pid: 30728, threadinfo=c987c000 task=c156abd0)
Stack: dacae2f4 00000000 c987cf74 c987cefc c987c000 c0221194 dfced400 c987cebc
       00000000 c0220b0a dfced400 c987cebc 00000000 c013e3af c154ce00 c987cebc
       c987cf14 c013e430 c154ce00 c987cebc d24a1001 58465342 00000000 bfeb0fdc
Call Trace:
 [<c0221194>] vfs_statvfs+0x34/0x38
 [<c0220b0a>] linvfs_statfs+0x28/0x2e
 [<c013e3af>] vfs_statfs+0x4b/0x66
 [<c013e430>] vfs_statfs64+0x23/0xb2
 [<c013e5ea>] sys_statfs64+0x81/0xbf
 [<c024fac4>] tty_write+0x179/0x1bc
 [<c0111811>] recalc_task_prio+0x93/0x188
 [<c0105878>] math_state_restore+0x28/0x42
 [<c0103aa1>] sysenter_past_esp+0x52/0x71
Code: 8b 00 c7 45 24 ff 00 00 00 89 c2 0f b6 c8 25 00 ff 0f 00 c1
 <1>Unable to handle kernel NULL pointer dereference at virtual address 
00000000
 printing eip:
c020ebb4
*pde = 00000000
Oops: 0000 [#4]
CPU:    0
EIP:    0060:[<c020ebb4>]    Not tainted
EFLAGS: 00010202   (2.6.8-rc4)
EIP is at xfs_statvfs+0xb6/0xe8
eax: 00000000   ebx: dacae314   ecx: dacae2f4   edx: 00000000
esi: dacae2f4   edi: 00000000   ebp: d27c1ebc   esp: d27c1e68
ds: 007b   es: 007b   ss: 0068
Process df (pid: 7037, threadinfo=d27c1000 task=c64726c0)
Stack: dacae2f4 00000000 d27c1f74 d27c1efc d27c1000 c0221194 dfced400 d27c1ebc
       00000000 c0220b0a dfced400 d27c1ebc 00000000 c013e3af c154ce00 d27c1ebc
       d27c1f14 c013e430 c154ce00 d27c1ebc d75b8001 58465342 00000000 bfeb0fdc
Call Trace:
 [<c0221194>] vfs_statvfs+0x34/0x38
 [<c0220b0a>] linvfs_statfs+0x28/0x2e
 [<c013e3af>] vfs_statfs+0x4b/0x66
 [<c013e430>] vfs_statfs64+0x23/0xb2
 [<c013e5ea>] sys_statfs64+0x81/0xbf
 [<c024fac4>] tty_write+0x179/0x1bc
 [<c0254921>] write_chan+0x0/0x219
 [<c024f94b>] tty_write+0x0/0x1bc
 [<c01403e9>] vfs_write+0xc9/0x119
 [<c014050a>] sys_write+0x51/0x80
 [<c0103aa1>] sysenter_past_esp+0x52/0x71
Code: 8b 00 c7 45 24 ff 00 00 00 89 c2 0f b6 c8 25 00 ff 0f 00 c1
[...]

XFS keeps segfaulting and dying in my machine. It is so strange...

General data:
x86 P-IV 2.5 GHz
512 MB RAM
The filesystem was onto a MAXTOR 70 GB SCSI disk, connected to Adaptec AIC79XX PCI-X SCSI card. The other filesystem containing files over XFS is a RAID0 over two identical IDE disks.

If you need further information, like .config or so, please do not hesitate
to ask.

Thanks in advance,

Ender.

 Why is a cow? Mu. (Ommmmmmmmmm)

Did /mnt run out of space while doing that? Or nearly? There's a known issue with that area of the XFS code, in conjunction with 4K stacks at the moment - was that enabled in your .config?

Looks like something stamped on parts of the xfs_mount structure for the filesystem mounted at /mnt, a stack overrun would explain that and your subsequent oopsen.

cheers.

Nathan

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello, Nathan, many thanks for the reply.

The machine locked up last night again, so I cannot login, and I cannot
manage to remember if I enabled 4K stacks in the code, could be possible, but I am not sure.

What I am sure of is that /mnt was plenty (8-10 GB or so) of disk space. I
had two or three concurrent accesses, IIRC, some of them writing from the RAID to the SCSI (both using XFS), and the last one finding and removing unwanted packages.

Could this scenario give you some hints? I will return where the server is in
5 hours, and I will send to the list the .config.

Again, many thanks,

Ender.
- --
Prepare ship for ludicrous speed! Fasten all seatbelts, seal all
entrances and exits, close all shops in the mall, cancel the three ring circus, secure all animals in the zoo!
-- Colonel Sandurz (Spaceballs).
- --
Servicios de red - Network services
RedIRIS - Spanish Academic Network for Research and Development Red.es - Madrid (Spain)
Tlf (+34) 91.212.76.25
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBJGc/Ws/EhA1iABsRAqYVAKDPWUIi6m5pBH3nwbgHttV9ko436ACfYDbp F+SsCIE95BvW9m7YMbg/yc8=
=MHxU
-----END PGP SIGNATURE-----

Hello, Nathan. I am here at the University now, and yes, the kernel were
compiled with 4K stacks. I rebuilt it with 8K and rebooted:

ulises:~# zcat /proc/config.gz |grep STACKS
# CONFIG_4KSTACKS is not set

I ran a disk test and waited.

After an hour, the system crashed. I forgot to plug in a console, so rebooted
and waited again. And here it is:

------------[ cut here ]------------
kernel BUG at mm/vmscan.c:565!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c01307f1>]    Not tainted
EFLAGS: 00010046   (2.6.8.1)

EIP is at shrink_cache+0x7c/0x2c7
eax: 00000000 ebx: c0415d24 ecx: c12485f8 edx: c12485f8
esi: 0000001d edi: c0415d48 ebp: 0000001c esp: c15bfeb0
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 10, threadinfo=c15be000 task=c15720b0)

Stack: c15ab080 00000002 c15be000 00000020 c10444f8 c10f8b18 00000000 00000001
       00000286 c15c0130 c15bfee4 c012ddd8 00000000 00000286 00000000 00000000
       00000000 00000000 c15bfef8 c15bfef8 00000000 00000000 00000000 dff28180
Call Trace:
 [<c012ddd8>] drain_cpu_caches+0x40/0x42
 [<c0130085>] shrink_slab+0x7b/0x186
 [<c0130f19>] shrink_zone+0x9e/0xb8
 [<c01312d6>] balance_pgdat+0x1c9/0x22d
 [<c0131401>] kswapd+0xc7/0xd7
 [<c0112dac>] autoremove_wake_function+0x0/0x57
 [<c0112dac>] autoremove_wake_function+0x0/0x57
 [<c013133a>] kswapd+0x0/0xd7
 [<c0101fdd>] kernel_thread_helper+0x5/0xb
Code: 0f 0b 35 02 02 81 3c c0 8b 51 04 8b 01 89 50 04 89 02 c7 41
 <1>Unable to handle kernel NULL pointer dereference at virtual address 

00000004
printing eip:
c012e3c7
*pde = 00000000

Oops: 0000 [#2]
CPU:    0
EIP:    0060:[<c012e3c7>]    Not tainted
EFLAGS: 00010006   (2.6.8.1)

EIP is at free_block+0x43/0xcb
eax: 00800000 ebx: 00000000 ecx: 00000000 edx: c1000000
esi: c0415d48 edi: 00000000 ebp: c0415d54 esp: c15bfb20
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 10, threadinfo=c15be000 task=c15720b0)

Stack: c022b4c2 dffe0090 c0415d64 2002002c d242f800 00000286 c1401480 c012e499
       c0415d48 c1161948 2002002c c1161948 c1161938 d242f800 00000286 0000062a
       c012e7a6 c0415d48 c1161938 c1a4ce80 dcdab000 00000004 c0338237 d242f800
Call Trace:
 [<c022b4c2>] memmove+0x4d/0x4f
 [<c012e499>] cache_flusharray+0x4a/0xb6
 [<c012e7a6>] kfree+0x5e/0x62
 [<c0338237>] kfree_skbmem+0x13/0x2c
 [<c03382b8>] __kfree_skb+0x68/0xdd
 [<c03591aa>] tcp_clean_rtx_queue+0x12f/0x3a0
 [<c0359a53>] tcp_ack+0xb4/0x560
 [<c035bc4d>] __tcp_data_snd_check+0xd3/0xe2
 [<c035c454>] tcp_rcv_established+0x419/0x84a
 [<c036470d>] tcp_v4_do_rcv+0x117/0x11c
 [<c0364d51>] tcp_v4_rcv+0x63f/0x884
 [<c025b3a5>] scrup+0xe3/0xf7
 [<c034badf>] ip_local_deliver+0xa3/0x1a2
 [<c034bec4>] ip_rcv+0x2e6/0x3fe
 [<c033d335>] netif_receive_skb+0x14b/0x17e
 [<c033d3dd>] process_backlog+0x75/0xf6
 [<c033d4c8>] net_rx_action+0x6a/0xe2
 [<c0117d52>] __do_softirq+0x7e/0x80
 [<c0117d7a>] do_softirq+0x26/0x28
 [<c0105e79>] do_IRQ+0xc4/0xdf
 [<c0104e82>] do_invalid_op+0x0/0xcb
 [<c0104460>] common_interrupt+0x18/0x20
 [<c0104e82>] do_invalid_op+0x0/0xcb
 [<c0104b61>] die+0x7a/0xcd
 [<c0104f4b>] do_invalid_op+0xc9/0xcb
 [<c01307f1>] shrink_cache+0x7c/0x2c7
 [<c02217da>] vn_purge+0x108/0x116
 [<c010451d>] error_code+0x2d/0x38
 [<c01307f1>] shrink_cache+0x7c/0x2c7
 [<c012ddd8>] drain_cpu_caches+0x40/0x42
 [<c0130085>] shrink_slab+0x7b/0x186
 [<c0130f19>] shrink_zone+0x9e/0xb8
 [<c01312d6>] balance_pgdat+0x1c9/0x22d
 [<c0131401>] kswapd+0xc7/0xd7
 [<c0112dac>] autoremove_wake_function+0x0/0x57
 [<c0112dac>] autoremove_wake_function+0x0/0x57
 [<c013133a>] kswapd+0x0/0xd7
 [<c0101fdd>] kernel_thread_helper+0x5/0xb
Code: 8b 53 04 8b 03 89 50 04 89 02 c7 43 04 00 02 20 00 2b 4b 0c
 <0>Kernel panic: Fatal exception in interrupt

In interrupt handler - not syncing

Last time I seem to remember some function related to ext3, so this is more a
data/stack corruption than others, what do you think?

I attach current dmesg and config, if it helps.

Is there anything else I could do? Patches, other trees, special
configurations? This machine seems impossible to stabilize.

The machine is using SiI libata driver, if it could have some problem
related...

Many thanks in advance,

Ender.

 Why is a cow? Mu. (Ommmmmmmmmm)

Hmmm, that looks awfully similar to the problem other folks are chasing (see thread "Possible dcache BUG"). Is it reproducible for you by any chance?

There have been some debugging patches that Marcelo posted in that thread, would be a good idea to try those out; although with your crash its difficult to see which slab it is thats corrupted (iirc, Marcelo's patch was targetting the buffer_head slab).

Nathan