Kernel crash with >2.6.28.10 + drbd + nfs + xfs + heartbeat

DISCUSSION Stuart Sheldon 2 participants 3 messages Aug 19, 2009

lore

Share & tools

lore

Stuart Sheldon

author

2009-08-19 17:37 UTC #0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

I've traced this down to what looks to be a XFS problem. When executing a hb_standby the system crashes with the following error on the console:

- ------ Start Errors --------
Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel:last sysfs file:
/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel:invalid opcode: 0000 [#1] SMP

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel:Stack:

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: ffff880217dfad10 ffff880217dfad00 ffff880217dfad10 ffffffffa034b8ea

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel:Call Trace:

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffffa034cfda>] ? xfs_inode_set_reclaim_tag+0x69/0x89 [xfs]

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: ffff88021197de58 0000000000000001 ffff88021197de58 ffffffffa033ff6e

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffffa033ff6e>] ? xfs_reclaim+0x99/0x9f [xfs]

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffffa034b8ea>] ? xfs_fs_destroy_inode+0x2f/0x4d [xfs]

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8029b034>] ? dispose_list+0xcd/0xfb

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8029b386>] ? invalidate_inodes+0xfe/0x11c

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8028c01f>] ? generic_shutdown_super+0x49/0xfd

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8028c0f5>] ? kill_block_super+0x22/0x3a

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8028c740>] ? deactivate_super+0x5f/0x77

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8029e62a>] ? sys_umount+0x2d8/0x307

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel: [<ffffffff8020b96b>] ? system_call_fastpath+0x16/0x1b

Message from syslogd@drbd-1 at Aug 19 09:30:43 ...
kernel:Code: 18 02 00 00 48 d3 e8 89 c1 83 e1 3f 41 0f a3 0c 11 19 c0
85 c0 75 07 49 8d 04 11 0f ab 08 48 63 c1 4d 8b 44 c0 18 4d 85 c0 75 04 <0f> 0b eb fe 41 83 eb 06 41 ff ca 45 85 d2 75 bc 4d 85 c0 74 16
- --------- End Errors --------

System info:
Intel(R) Celeron(R) CPU E1400 @ 2.00GHz

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm

Kernel: Vanilla 2.6.30.5 amd64 (anything past 2.6.28.10 will produce problem)

Distro: Debian Lenny

XFS is built as a module. Here is the Heartbeat command file that creams the machine:

drbd-1 IPaddr::xxx.xxx.xxx.xxx/26/eth0 \
        drbddisk::r0 \
        Filesystem::/dev/drbd0::/data::xfs::noatime,nodiratime \
        killnfsd \
        nfs-common \
        nfs-kernel-server \
        MailTo::root::NFS_Fallover_Notice

I can pretty much duplicate this at will. I just need to slot a maintenance window.

Let me know if you need any more info... I'd really like to help get this resolved.

Thanks in advance.

Stuart Sheldon
ACT USA

- --
And so it was, that later, as the miller told his tale, that her face at first just ghostly, turned a whiter shade of pale...
-- Procol Harum - "Whiter Shade of Pale Lyrics"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQIcBAEBCAAGBQJKjDfzAAoJEFKVLITDJSGSuCcP/i0noYmVN4s1dVRzyDuXXLoM nIt/cdvMZM2KjkrYlDtRZcd3HF3SyDQX024f0hbncvPQrVBcgEeH18WJpXPTLHw1 S47ETIZHwVeI8Jm174rl/uVHZS9YyPlUMysNgOhdT+CZYLIxWCzVXWLenSdL5n7O YaFPEJRKwNvX8iui9DaWkXJdEmRM0LADDkMO4MtrclBeS6GP8+DMgIBJb0mKNkie y6L67lUiqm6QYaUaGA4hotQJ5IKUbxxRGllil9TB3goQ+lfeUBgBwmKEdPIWb+DV 8V5LG3/o2gbfPSWksmTuk2+A2M/6g3x/2btfvOlS8/DX3lMzTEdDIdM4+YlK9Uvf drKLvKGwAfaqBwfE45ax6Kc7deOWGTAKD6zxXtVIvryGit+aHSJ9Ygf6BdTzK/nA qa4+sa7t669bcg2YJztjinrsrwHnSroZCM5a6Nrg8geKt/WlBDXdykYtXhWnvF0s zulVLWkFhv4vnkjVi8Dx0SvpkII6T7b4x/ldMvdegxQWP0uzME4Ek6W4BwRzV/mQ +pi7X+Nvgc/zoEQEpmj62dIzxr8NS927pR+dmG8xF6z1VcYsyfxUPxMXCs4uMW4m IfzqBC9E48NyTkHmaCyR6yNTA7HtkEtFeTm/6Gj4C3jAV5DylfmMVsyO1cOdhj7y JlLJD0Oe8NGORnjt3Xs9
=zccl
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Christoph Hellwig

2009-08-19 18:23 UTC #1 lore raw

Quoted 7 lines · Stuart Sheldon

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi all,
I've traced this down to what looks to be a XFS problem. When executing a hb_standby the system crashes with the following error on the console:

Known problem. Fixed in latest mainline, and
http://bugzilla.kernel.org/show_bug.cgi?id=13375 has backports for 2.6.29 and 2.6.30. I will send them to 2.6.30-stable this week, but there is no 2.6.29-stable anymore.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Stuart Sheldon

author

2009-08-19 18:35 UTC #2 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Thanks!

I've patched our tree and submitted the build job. Will report back with test results.

Stu

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Stuart Sheldon
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi all,

Quoted 8 lines

Quoted 2 lines
I've traced this down to what looks to be a XFS problem. When executing a hb_standby the system crashes with the following error on the console:
Known problem. Fixed in latest mainline, and
http://bugzilla.kernel.org/show_bug.cgi?id=13375 has backports for 2.6.29 and 2.6.30. I will send them to 2.6.30-stable this week, but there is no 2.6.29-stable anymore.

- --
And you run and you run to catch up with the sun, but its sinking And racing around to come up behind you again
The sun is the same in the relative way, but youre older Shorter of breath and one day closer to death
-- Pink Floyd - "Time Lyrics"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQIcBAEBCAAGBQJKjEX0AAoJEFKVLITDJSGSXxgP/0qSGvIf/WHUm6txOYRo7lAH xALV13fpfIi4ymlOtRztAFcSk4U2ciY3jgewfCK6/41EHwaeWkt3lCDPP8lLdgsA PnZNBRPvR9vj5WxI7542VMuLgoBacCFo5xC6i8gE4ftCxmHLKH3GgLPTtzjjJ9AF QtR+Maj40R3zOPQRJ94xgeTS9ZAO0DNWxf/ozqQqPdFPitqO9MXfPQibwLmQhz+a n5fXgHV5pBymDnvY2YCBTTBX16dG8kvrnQrFuLVTmFRvco4A20V73sPnszMULQpn rUb5n5C1KZuXqFj+JOQB7bcMCvK6IQ4d6lQkSwie/jVsBMtNCTTxWd7Sl5KsbjJg ELG9M2aINkQG95s07F+Y0be40KKkrrr/5hp6Hb2i+zXTjjBIspy9GSZpTJVL557r jjDwoHPHvkAXasi2iC2Yqqn03vUQq/uzv7QoLf3mrdUNY3lLjiEEU1Cyt8CJO8Gx +t3OZIi7tQhy6QlgNuZGiX6CKW9HdpITBpv1JabOaqmlKHB6jQICJiYjGVOf0lkW NMIVzK6rm8TAAcvQhFsBN9wrdJebpuPfyRFij4z5GfMLppxq0GT+MWw84/UKbxrE NJHHkvrfDvLgWe4jx7gviJr/CNH3UT579Jjy024hUFBXBTUKSbjMoOt1+9W6njIu BzIBzS+Rt8Ujtg32qgjY
=CePG
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs