XFS hang during xfs_fsr run

Hi XFS-List!
We recently had two hangs on one of our servers, which seem to be related to XFS. We managed to capture the dmesg output before hard-resetting the machine. It seems that those hangs are caused by xfs_fsr which was running at that time. This is a mail server with millions of files. The underlying file system was already checked with xfs_repair without finding errors. Is there anything i could try to rectify or at least narrow down this problem?

******** Trace 1 start ********

[169342.414517] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
[169342.414646] IP: [<ffffffff811abd75>] 0xffffffff811abd75
[169342.414651] PGD 26fba0067 PUD 2935aa067 PMD 0
[169342.414655] Oops: 0000 [#1] SMP
[169342.414658] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:06:00.0/0000:07:00.0/0000:08:00.0/0000:09:00.0/irq
[169342.414663] CPU 2
[169342.414668] Pid: 23782, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
[169342.414671] RIP: 0010:[<ffffffff811abd75>]  [<ffffffff811abd75>] 0xffffffff811abd75
[169342.414675] RSP: 0018:ffff88001eb2db78  EFLAGS: 00010296
[169342.414678] RAX: 0000000000000008 RBX: ffff880044c46bc0 RCX: ffff88001eb2dd54
[169342.414681] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff880044c46bc0
[169342.414684] RBP: ffff88001eb2dba8 R08: 0000000000000000 R09: ffff88032fb3ec00
[169342.414687] R10: ffff88001eb2d9e8 R11: ffffffff811eebc1 R12: ffff88008d17a400
[169342.414690] R13: 0000000000000005 R14: 0000000000000000 R15: ffff88008d17a438
[169342.414694] FS:  00007f760bcd66f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[169342.414697] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[169342.414700] CR2: 0000000000000018 CR3: 000000029347c000 CR4: 00000000000006e0
[169342.414703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[169342.414706] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[169342.414709] Process xfs_fsr (pid: 23782, threadinfo ffff88001eb2c000, task ffff88032de43cc0)
[169342.414711] Stack:
[169342.414713]  ffff88001eb2dba8 ffffffff811abc98 ffff88008d17a400 0000000000000000
[169342.414717] <0> ffff88008d17a400 ffffffffffffffff ffff88001eb2dce8 ffffffff8117ec2c
[169342.414721] <0> ffff88001eb2dca8 0000000000000000 ffff880100000000 ffff880300000000
[169342.414726] Call Trace:
[169342.414731]  [<ffffffff811abc98>] ? 0xffffffff811abc98
[169342.414734]  [<ffffffff8117ec2c>] 0xffffffff8117ec2c
[169342.414738]  [<ffffffff81198b2b>] 0xffffffff81198b2b
[169342.414741]  [<ffffffff811af419>] 0xffffffff811af419
[169342.414743]  [<ffffffff813629e8>] ? 0xffffffff813629e8
[169342.414746]  [<ffffffff811b976e>] 0xffffffff811b976e
[169342.414749]  [<ffffffff810dd9ea>] 0xffffffff810dd9ea
[169342.414751]  [<ffffffff810de158>] 0xffffffff810de158
[169342.414754]  [<ffffffff810de1e4>] 0xffffffff810de1e4
[169342.414757]  [<ffffffff810dd22e>] 0xffffffff810dd22e
[169342.414759]  [<ffffffff810da685>] 0xffffffff810da685
[169342.414762]  [<ffffffff810da76f>] 0xffffffff810da76f
[169342.414765]  [<ffffffff810daebf>] 0xffffffff810daebf
[169342.414767]  [<ffffffff810cc1b6>] 0xffffffff810cc1b6
[169342.414770]  [<ffffffff810cc1f3>] 0xffffffff810cc1f3
[169342.414773]  [<ffffffff810c9396>] 0xffffffff810c9396
[169342.414775]  [<ffffffff810c943a>] 0xffffffff810c943a
[169342.414778]  [<ffffffff810029ab>] 0xffffffff810029ab
[169342.414780] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36 48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90 90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[169342.414805] RIP  [<ffffffff811abd75>] 0xffffffff811abd75
[169342.414808]  RSP <ffff88001eb2db78>
[169342.414810] CR2: 0000000000000018
[169342.414813] ---[ end trace 5e5d73a1b2a79389 ]---
[169434.738237] ------------[ cut here ]------------
[169434.738315] kernel BUG at fs/xfs/xfs_iget.c:295!
[169434.738387] invalid opcode: 0000 [#2] SMP
[169434.738553] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[169434.738644] CPU 2
[169434.738761] Pid: 23899, comm: mktemp Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
[169434.738854] RIP: 0010:[<ffffffff81196309>]  [<ffffffff81196309>] 0xffffffff81196309
[169434.738994] RSP: 0018:ffff88014ac5da18  EFLAGS: 00010246
[169434.739068] RAX: 0000000000000000 RBX: ffff880044c46260 RCX: ffffffff811f2d94
[169434.739127] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff88008d17a4e4
[169434.739127] RBP: ffff88014ac5dab8 R08: ffff88002828fe10 R09: ffff88014ac5d8e8
[169434.739127] R10: ffff88014ac5d948 R11: ffff88008d17a458 R12: ffff88008d17a458
[169434.739127] R13: 0000000000000002 R14: ffff88032dac6ba8 R15: 0000000000000017
[169434.739127] FS:  00007fc88a5266f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[169434.739127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

******** Trace 1 end ********

******** Trace 2 start ********

[40146.682000] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
[40146.682192] IP: [<ffffffff811abd75>] 0xffffffff811abd75
[40146.682314] PGD 236381067 PUD 2883d3067 PMD 0
[40146.682528] Oops: 0000 [#1] SMP
[40146.682692] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[40146.682784] CPU 3
[40146.682901] Pid: 19417, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
[40146.682901] RIP: 0010:[<ffffffff811abd75>]  [<ffffffff811abd75>] 0xffffffff811abd75
[40146.682901] RSP: 0018:ffff88025c721b78  EFLAGS: 00010296
[40146.682901] RAX: 0000000000000008 RBX: ffff8802686d4320 RCX: ffff88025c721d54
[40146.682901] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff8802686d4320
[40146.682901] RBP: ffff88025c721ba8 R08: 000000000000007c R09: ffff88032fb1ac00
[40146.682901] R10: ffff88025c721a28 R11: 0000000000000296 R12: ffff88022ac2c400
[40146.682901] R13: 0000000000000005 R14: 0000000000000000 R15: ffff88022ac2c438
[40146.682901] FS:  00007f5a735786f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
[40146.682901] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[40146.682901] CR2: 0000000000000018 CR3: 000000029b82e000 CR4: 00000000000006e0
[40146.682901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[40146.682901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[40146.682901] Process xfs_fsr (pid: 19417, threadinfo ffff88025c720000, task ffff88032d2b0000)
[40146.682901] Stack:
[40146.682901]  ffff88025c721ba8 ffffffff811abc98 ffff88022ac2c400 0000000000000000
[40146.682901] <0> ffff88022ac2c400 ffffffffffffffff ffff88025c721ce8 ffffffff8117ec2c
[40146.682901] <0> ffff88025c721ca8 0000000000000000 ffff880200000000 0000000700000000
[40146.682901] Call Trace:
[40146.682901]  [<ffffffff811abc98>] ? 0xffffffff811abc98
[40146.682901]  [<ffffffff8117ec2c>] 0xffffffff8117ec2c
[40146.682901]  [<ffffffff81198b2b>] 0xffffffff81198b2b
[40146.682901]  [<ffffffff811af419>] 0xffffffff811af419
[40146.682901]  [<ffffffff813629e8>] ? 0xffffffff813629e8
[40146.682901]  [<ffffffff811b976e>] 0xffffffff811b976e
[40146.682901]  [<ffffffff810dd9ea>] 0xffffffff810dd9ea
[40146.682901]  [<ffffffff810de158>] 0xffffffff810de158
[40146.682901]  [<ffffffff810de1e4>] 0xffffffff810de1e4
[40146.682901]  [<ffffffff810dd22e>] 0xffffffff810dd22e
[40146.682901]  [<ffffffff810da685>] 0xffffffff810da685
[40146.682901]  [<ffffffff810da76f>] 0xffffffff810da76f
[40146.682901]  [<ffffffff810daebf>] 0xffffffff810daebf
[40146.682901]  [<ffffffff810cc1b6>] 0xffffffff810cc1b6
[40146.682901]  [<ffffffff810cc1f3>] 0xffffffff810cc1f3
[40146.682901]  [<ffffffff810c9396>] 0xffffffff810c9396
[40146.682901]  [<ffffffff810c943a>] 0xffffffff810c943a
[40146.682901]  [<ffffffff810029ab>] 0xffffffff810029ab
[40146.682901] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36 48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90 90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[40146.682901] RIP  [<ffffffff811abd75>] 0xffffffff811abd75
[40146.682901]  RSP <ffff88025c721b78>
[40146.682901] CR2: 0000000000000018
[40146.689961] ---[ end trace 53b9544a53a60243 ]---

******** Trace 2 end ********

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Can you run the trace through ksymoops so that we get function names rather than numbers that don't mean anything?

Also, can you tell us what kernel version and distro you are running, as well as the version of xfs_fsr? These will help tell us if it is a new or known problem.

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Dave!

I've never done that before. Could you point me to an howto about this?

The Kernel version is 2.6.33 vanilla (self-compiled), Distro is Gentoo Linux, xfs_fsr Version is 3.1.1 (also self-compiled).

Please let me know if you need anything else.

thanks,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

I've never needed to use it myself, so all I'd be doing is reading the man page and doing a google search to answer your question. It's way past my bed time here, so I'll leave that to you for the moment - try searching for "ksymoops howto"....

Ok, so it's all pretty recent then.

Was there anything else in the logs prior to the oops messages that might indicate errors were occurring?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Also when you next rebuilt the kernel please make sure to include CONFIG_KALLSYMS in the configuration, possibly CONFIG_KALLSYMS_ALL too. This will help greatly with decoding any kind of warning / oops.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Christoph/Dave!

Thanks for this information. Unfortunately my current kernel was built without CONFIG_KALLSYMS. I'm now recompiling with CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL set. I reckon that my old traces can't be ksymoops'ed even if i enable that kernel option now? I will see if i can get a fresh trace then (even though i hope it won't happen again).

Unfortunately everything in the logs is dandy until the error happens. It seems that xfs_fsr randomly stops at some files and then locks up the whole /var partition. I searched for the inode numbers where xfs_fsr stopped and one time it was "/var/log/xfs_fsr.log" and the other time it was "/var/spool/imap/x/user/xxxx/cyrus.cache" (username obfuscated). Whats's interesting is that i have the no-defrag flag set on the whole /var/log directory and still it seemed to hang on that log file.

cheers,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

There's an old userspace ksymoops tool which we used before kallysyms was around. I don't remember the exact usage, just that it was a real pain.

What no-defrag flag?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi!

After some research on how to do a proper ksymoops i found the following information in Documentation/oops-tracing.txt:
NOTE: ksymoops is useless on 2.6. Please use the Oops in its original format
(from dmesg, etc). Ignore any references in this or other docs to "decoding
the Oops" or "running it through ksymoops". If you post an Oops from 2.6 that
has been run through ksymoops, people will just tell you to repost it.

What should i do now? I've already enabled CONFIG_KALLSYMS and rebooted.

# xfs_io /var/log/
xfs_io> chattr -R +f
xfs_io> lsattr
------------f- /var/log/

cheers,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Yeah, that seems to be the case.

If you've got the inode numbers, then your running with the verbose flag set? Do you still have the logs for those inodes that it hung on?

xfs_fsr doesn't do directory traversals to find files for defrag - it uses more efficient bulkstat+open-by-handle method to visit every inode in the filesystem once. As a result, it will still open inodes that have the nodefrag flag set on them, but will then ignore them once it finds the flag is set.

If xfs_fsr hung before it checked the nodefrag flag, then there's only a few things it could get stuck on:

1. fsync() of the file
2. file lock checks
3. statvfs64()
4. ioctl(XFS_IOC_FSGETXATTR)

A trace would tell us which one it was....

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Yes I am running xfs_fsr with the -v flag. But I'm not 100% sure if the log was truncated since it resides in /var which locked up. Here are the last log entries before the oopses:
ino=134269708 (this inode was /var/log/xfs_fsr.log)
(hang)
...
ino=277040401 (this inode was /var/spool/imap/x/user/xxxx/cyrus.cache)
(hang)

I can pin down those things at the moment:
- Both times it was a "hot" file - in one case a logfile, in the other case a database file.
- Usually it should say "file busy" and continue but sometimes it doesn't and just hangs the filesystem + oopses the kernel.
- It happens randomly, if i rerun xfs_fsr after the hang it usually goes over the "problem" file without a hickup.
- On /var/log/xfs_fsr.log it hung even though the (+f) chattr was set.

Yes, usually it says "marked as don't defrag, ignoring" but this one time it hung.

Will see that i can get another one.

cheers,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Got another one, this time with ksyms enabled:

[192115.749003] BUG: unable to handle kernel NULL pointer dereference at
0000000000000018
[192115.749197] IP: [<ffffffff811ad69d>] xfs_trans_find_item+0x1/0xa
[192115.749259] PGD 2f0dec067 PUD 1b1ab9067 PMD 0
[192115.749264] Oops: 0000 [#1] SMP
[192115.749267] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[192115.749271] CPU 3
[192115.749276] Pid: 1671, comm: xfs_fsr Not tainted 2.6.33 #1
0JR815/PowerEdge 2950
[192115.749279] RIP: 0010:[<ffffffff811ad69d>]  [<ffffffff811ad69d>]
xfs_trans_find_item+0x1/0xa
[192115.749284] RSP: 0018:ffff8800aec13b78  EFLAGS: 00010296
[192115.749287] RAX: 0000000000000008 RBX: ffff8800ad600320 RCX:
ffff8800aec13d54
[192115.749289] RDX: 0000000000000005 RSI: 0000000000000000 RDI:
ffff8800ad600320
[192115.749292] RBP: ffff8800aec13ba8 R08: 000000000000007c R09:
ffff88032fb0b600
[192115.749295] R10: 00000000ffffffff R11: ffff8800282cde60 R12:
ffff8802d3f12c00
[192115.749298] R13: 0000000000000005 R14: 0000000000000000 R15:
ffff8802d3f12c38
[192115.749302] FS:  00007f8f3cc5a6f0(0000) GS:ffff8800282c0000(0000)
knlGS:0000000000000000
[192115.749305] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[192115.749307] CR2: 0000000000000018 CR3: 000000014904b000 CR4:
00000000000006e0
[192115.749310] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[192115.749313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[192115.749316] Process xfs_fsr (pid: 1671, threadinfo ffff8800aec12000,
task ffff88032e77b600)
[192115.749318] Stack:
[192115.749320]  ffff8800aec13ba8 ffffffff811ad5c0 ffff8802d3f12c00
0000000000000000
[192115.749324] <0> ffff8802d3f12c00 ffffffffffffffff ffff8800aec13ce8
ffffffff81180554
[192115.749328] <0> ffff8800aec13ca8 0000000000000000 ffff880000000000
ffff880300000000
[192115.749332] Call Trace:
[192115.749338]  [<ffffffff811ad5c0>] ? xfs_trans_log_inode+0x22/0x4c
[192115.749344]  [<ffffffff81180554>] xfs_bunmapi+0x9ec/0xa36
[192115.749350]  [<ffffffff8119a452>] xfs_itruncate_finish+0x188/0x2db
[192115.749355]  [<ffffffff811b0d41>] xfs_inactive+0x218/0x435
[192115.749360]  [<ffffffff81364338>] ? __mutex_lock_slowpath+0x22d/0x23c
[192115.749365]  [<ffffffff811bb096>] xfs_fs_clear_inode+0xb3/0xb8
[192115.749371]  [<ffffffff810df28a>] clear_inode+0x78/0xd1
[192115.749375]  [<ffffffff810df9f8>] generic_delete_inode+0xf6/0x16b
[192115.749379]  [<ffffffff810dfa84>] generic_drop_inode+0x17/0x62
[192115.749382]  [<ffffffff810deace>] iput+0x61/0x65
[192115.749386]  [<ffffffff810dbf25>] dentry_iput+0xb5/0xc5
[192115.749389]  [<ffffffff810dc00f>] d_kill+0x43/0x63
[192115.749393]  [<ffffffff810dc75f>] dput+0x148/0x155
[192115.749398]  [<ffffffff810cda56>] __fput+0x196/0x1bb
[192115.749401]  [<ffffffff810cda93>] fput+0x18/0x1a
[192115.749405]  [<ffffffff810cac36>] filp_close+0x67/0x72
[192115.749409]  [<ffffffff810cacda>] sys_close+0x99/0xd2
[192115.749415]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
[192115.749417] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36
48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90
90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[192115.749441] RIP  [<ffffffff811ad69d>] xfs_trans_find_item+0x1/0xa
[192115.749445]  RSP <ffff8800aec13b78>
[192115.749447] CR2: 0000000000000018
[192115.749458] ---[ end trace b5b512ceb91c24c7 ]---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

...

That's ... unexpected. That implies that ip->i_temp == NULL after it has been joined to a transaction. I can't see how that could occur there. Can you recompile the kernel with CONFIG_XFS_DEBUG and re-run the test as that option includes all sorts of sanity checks for ip->i_temp?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Dave!

CONFIG_XFS_DEBUG is enabled now. Is there anything special in dmesg i should look for? Or should i just send another oops when it happens again?

thanks,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Odds are you'll still oops, just earlier. Send anything interesting you see in the dmesg if/when it happens...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Oops is attached. This time the system kept working but hat a constant load of >2.0 after the incident. So i rebooted it afterwards.

Mar  8 22:53:54 email kernel: [52982.051367] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:53:54 email kernel: [52982.051420] ------------[ cut here ]------------
Mar  8 22:53:54 email kernel: [52982.051494] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:53:54 email kernel: [52982.051567] invalid opcode: 0000 [#1] SMP 
Mar  8 22:53:54 email kernel: [52982.051732] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:53:54 email kernel: [52982.051823] CPU 2 
Mar  8 22:53:54 email kernel: [52982.051940] Pid: 14867, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:53:54 email kernel: [52982.052031] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:53:54 email kernel: [52982.052127] RSP: 0018:ffff8801bd0b9c18  EFLAGS: 00010296
Mar  8 22:53:54 email kernel: [52982.052127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011e5f9
Mar  8 22:53:54 email kernel: [52982.052127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:53:54 email kernel: [52982.052127] RBP: ffff8801bd0b9c18 R08: 0000000000000002 R09: ffff8801bd0b9a68
Mar  8 22:53:54 email kernel: [52982.052127] R10: 000000000000000c R11: 0000000000018600 R12: 0000000000000002
Mar  8 22:53:54 email kernel: [52982.052127] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801bd0b9cf8
Mar  8 22:53:54 email kernel: [52982.052127] FS:  00007f0b5cee16f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  8 22:53:54 email kernel: [52982.052127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:53:54 email kernel: [52982.052127] CR2: 00007f0b5ceeb000 CR3: 0000000288306000 CR4: 00000000000006e0
Mar  8 22:53:54 email kernel: [52982.052127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:53:54 email kernel: [52982.052127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:53:54 email kernel: [52982.052127] Process xfs_fsr (pid: 14867, threadinfo ffff8801bd0b8000, task ffff8802c07b5100)
Mar  8 22:53:54 email kernel: [52982.052127] Stack:
Mar  8 22:53:54 email kernel: [52982.052127]  ffff8801bd0b9ce8 ffffffff8118af48 ffff8801bd0b9c58 ffffffff810d46cb
Mar  8 22:53:54 email kernel: [52982.052127] <0> ffff8801bd0b9dc8 ffff88032f6daa80 ffff8801bd0b9dc8 ffff88032c858190
Mar  8 22:53:54 email kernel: [52982.052127] <0> ffffffff811cf73e ffff88032d36e000 ffff8801bd0b9cb8 0000000000000018
Mar  8 22:53:54 email kernel: [52982.052127] Call Trace:
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:53:54 email kernel: [52982.052127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:53:54 email kernel: [52982.052127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:53:54 email kernel: [52982.052127]  RSP <ffff8801bd0b9c18>
Mar  8 22:53:54 email kernel: [52982.058596] ---[ end trace c02651fded7c4a28 ]---
Mar  8 22:54:03 email kernel: [52991.049266] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:03 email kernel: [52991.049413] ------------[ cut here ]------------
Mar  8 22:54:03 email kernel: [52991.049484] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:03 email kernel: [52991.049557] invalid opcode: 0000 [#2] SMP 
Mar  8 22:54:03 email kernel: [52991.049722] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:03 email kernel: [52991.049814] CPU 3 
Mar  8 22:54:03 email kernel: [52991.049930] Pid: 14870, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:03 email kernel: [52991.050023] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:03 email kernel: [52991.050169] RSP: 0018:ffff8801dab93c18  EFLAGS: 00010296
Mar  8 22:54:03 email kernel: [52991.050190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011f211
Mar  8 22:54:03 email kernel: [52991.050190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:03 email kernel: [52991.050190] RBP: ffff8801dab93c18 R08: 0000000000000000 R09: ffff8801dab93a68
Mar  8 22:54:03 email kernel: [52991.050190] R10: ffff88025ab93ab7 R11: ffff8801dab93978 R12: 0000000000000002
Mar  8 22:54:03 email kernel: [52991.050190] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801dab93cf8
Mar  8 22:54:03 email kernel: [52991.050190] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:03 email kernel: [52991.050190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:03 email kernel: [52991.050190] CR2: 00007f0b5ceeb000 CR3: 00000001d4cb2000 CR4: 00000000000006e0
Mar  8 22:54:03 email kernel: [52991.050190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:03 email kernel: [52991.050190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:03 email kernel: [52991.050190] Process xfs_fsr (pid: 14870, threadinfo ffff8801dab92000, task ffff8802c07b21c0)
Mar  8 22:54:03 email kernel: [52991.050190] Stack:
Mar  8 22:54:03 email kernel: [52991.050190]  ffff8801dab93ce8 ffffffff8118af48 ffff8801dab93c58 ffffffff810d46cb
Mar  8 22:54:03 email kernel: [52991.050190] <0> ffff8801dab93dc8 ffff88032f6daa80 ffff8801dab93dc8 ffff88032c858190
Mar  8 22:54:03 email kernel: [52991.050190] <0> ffffffff811cf73e ffff8801c6f18000 ffff8801dab93cb8 0000000000000018
Mar  8 22:54:03 email kernel: [52991.050190] Call Trace:
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:03 email kernel: [52991.050190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:03 email kernel: [52991.054038] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:03 email kernel: [52991.054038]  RSP <ffff8801dab93c18>
Mar  8 22:54:03 email kernel: [52991.054088] ---[ end trace c02651fded7c4a29 ]---
Mar  8 22:54:12 email kernel: [53000.103608] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:12 email kernel: [53000.103754] ------------[ cut here ]------------
Mar  8 22:54:12 email kernel: [53000.103825] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:12 email kernel: [53000.103898] invalid opcode: 0000 [#3] SMP 
Mar  8 22:54:12 email kernel: [53000.104063] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:12 email kernel: [53000.104127] CPU 2 
Mar  8 22:54:12 email kernel: [53000.104127] Pid: 14876, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:12 email kernel: [53000.104127] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:12 email kernel: [53000.104127] RSP: 0018:ffff8801f8001c18  EFLAGS: 00010296
Mar  8 22:54:12 email kernel: [53000.104127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011fe32
Mar  8 22:54:12 email kernel: [53000.104127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:12 email kernel: [53000.104127] RBP: ffff8801f8001c18 R08: 0000000000000000 R09: ffff8801f8001a68
Mar  8 22:54:12 email kernel: [53000.104127] R10: ffff880278001ab7 R11: ffff8801f8001978 R12: 0000000000000002
Mar  8 22:54:12 email kernel: [53000.104127] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801f8001cf8
Mar  8 22:54:12 email kernel: [53000.104127] FS:  00007f0b5cee16f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  8 22:54:12 email kernel: [53000.104127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:12 email kernel: [53000.104127] CR2: 00007f0b5ceeb000 CR3: 00000001e89bc000 CR4: 00000000000006e0
Mar  8 22:54:12 email kernel: [53000.104127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:12 email kernel: [53000.104127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:12 email kernel: [53000.104127] Process xfs_fsr (pid: 14876, threadinfo ffff8801f8000000, task ffff8802c07b6c00)
Mar  8 22:54:12 email kernel: [53000.104127] Stack:
Mar  8 22:54:12 email kernel: [53000.104127]  ffff8801f8001ce8 ffffffff8118af48 ffff8801f8001c58 ffffffff810d46cb
Mar  8 22:54:12 email kernel: [53000.104127] <0> ffff8801f8001dc8 ffff88032f6daa80 ffff8801f8001dc8 ffff88032c858190
Mar  8 22:54:12 email kernel: [53000.104127] <0> ffffffff811cf73e ffff88032d36c000 ffff8801f8001cb8 0000000000000018
Mar  8 22:54:12 email kernel: [53000.104127] Call Trace:
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:12 email kernel: [53000.104127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:12 email kernel: [53000.104127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:12 email kernel: [53000.104127]  RSP <ffff8801f8001c18>
Mar  8 22:54:12 email kernel: [53000.117360] ---[ end trace c02651fded7c4a2a ]---
Mar  8 22:54:21 email kernel: [53009.285360] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:21 email kernel: [53009.285506] ------------[ cut here ]------------
Mar  8 22:54:21 email kernel: [53009.285577] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:21 email kernel: [53009.285650] invalid opcode: 0000 [#4] SMP 
Mar  8 22:54:21 email kernel: [53009.285815] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:21 email kernel: [53009.285906] CPU 3 
Mar  8 22:54:21 email kernel: [53009.286023] Pid: 14883, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:21 email kernel: [53009.286115] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:21 email kernel: [53009.286189] RSP: 0018:ffff8801cdd99c18  EFLAGS: 00010296
Mar  8 22:54:21 email kernel: [53009.286189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000120a53
Mar  8 22:54:21 email kernel: [53009.286189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:21 email kernel: [53009.286189] RBP: ffff8801cdd99c18 R08: 0000000000000000 R09: ffff8801cdd99a68
Mar  8 22:54:21 email kernel: [53009.286189] R10: ffff88024dd99ab7 R11: ffff8801cdd99978 R12: 0000000000000002
Mar  8 22:54:21 email kernel: [53009.286189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801cdd99cf8
Mar  8 22:54:21 email kernel: [53009.286189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:21 email kernel: [53009.286189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:21 email kernel: [53009.286189] CR2: 00007f0b5ceeb000 CR3: 00000001d1bb6000 CR4: 00000000000006e0
Mar  8 22:54:21 email kernel: [53009.286189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:21 email kernel: [53009.286189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:21 email kernel: [53009.286189] Process xfs_fsr (pid: 14883, threadinfo ffff8801cdd98000, task ffff8802c07b21c0)
Mar  8 22:54:21 email kernel: [53009.286189] Stack:
Mar  8 22:54:21 email kernel: [53009.286189]  ffff8801cdd99ce8 ffffffff8118af48 ffff8801cdd99c58 ffffffff810d46cb
Mar  8 22:54:21 email kernel: [53009.286189] <0> ffff8801cdd99dc8 ffff88032f6daa80 ffff8801cdd99dc8 ffff88032c858190
Mar  8 22:54:21 email kernel: [53009.286189] <0> ffffffff811cf73e ffff8801c6f1a000 ffff8801cdd99cb8 0000000000000018
Mar  8 22:54:21 email kernel: [53009.286189] Call Trace:
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:21 email kernel: [53009.288582] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:21 email kernel: [53009.288582] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:21 email kernel: [53009.288582]  RSP <ffff8801cdd99c18>
Mar  8 22:54:21 email kernel: [53009.288673] ---[ end trace c02651fded7c4a2b ]---
Mar  8 22:54:30 email kernel: [53017.872753] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:30 email kernel: [53017.872900] ------------[ cut here ]------------
Mar  8 22:54:30 email kernel: [53017.872971] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:30 email kernel: [53017.873044] invalid opcode: 0000 [#5] SMP 
Mar  8 22:54:30 email kernel: [53017.873189] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:10:0d.0/local_cpus
Mar  8 22:54:30 email kernel: [53017.873189] CPU 3 
Mar  8 22:54:30 email kernel: [53017.873189] Pid: 14884, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:30 email kernel: [53017.873189] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:30 email kernel: [53017.873189] RSP: 0018:ffff8801c6ec5c18  EFLAGS: 00010296
Mar  8 22:54:30 email kernel: [53017.873189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000121674
Mar  8 22:54:30 email kernel: [53017.873189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:30 email kernel: [53017.873189] RBP: ffff8801c6ec5c18 R08: 0000000000000000 R09: ffff8801c6ec5a68
Mar  8 22:54:30 email kernel: [53017.873189] R10: ffff880246ec5ab7 R11: ffff8801c6ec5978 R12: 0000000000000002
Mar  8 22:54:30 email kernel: [53017.873189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801c6ec5cf8
Mar  8 22:54:30 email kernel: [53017.873189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:30 email kernel: [53017.873189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:30 email kernel: [53017.873189] CR2: 00007f0b5ceeb000 CR3: 00000001c6f10000 CR4: 00000000000006e0
Mar  8 22:54:30 email kernel: [53017.873189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:30 email kernel: [53017.873189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:30 email kernel: [53017.873189] Process xfs_fsr (pid: 14884, threadinfo ffff8801c6ec4000, task ffff8802c07b6c00)
Mar  8 22:54:30 email kernel: [53017.873189] Stack:
Mar  8 22:54:30 email kernel: [53017.873189]  ffff8801c6ec5ce8 ffffffff8118af48 ffff8801c6ec5c58 ffffffff810d46cb
Mar  8 22:54:30 email kernel: [53017.873189] <0> ffff8801c6ec5dc8 ffff88032f6daa80 ffff8801c6ec5dc8 ffff88032c858190
Mar  8 22:54:30 email kernel: [53017.873189] <0> ffffffff811cf73e ffff8801c6f1c000 ffff8801c6ec5cb8 0000000000000018
Mar  8 22:54:30 email kernel: [53017.873189] Call Trace:
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:30 email kernel: [53017.873189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:30 email kernel: [53017.873189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:30 email kernel: [53017.873189]  RSP <ffff8801c6ec5c18>
Mar  8 22:54:30 email kernel: [53017.880066] ---[ end trace c02651fded7c4a2c ]---
Mar  8 22:54:37 email kernel: [53025.709235] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:37 email kernel: [53025.709381] ------------[ cut here ]------------
Mar  8 22:54:37 email kernel: [53025.709452] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:37 email kernel: [53025.709525] invalid opcode: 0000 [#6] SMP 
Mar  8 22:54:37 email kernel: [53025.709690] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:37 email kernel: [53025.709782] CPU 3 
Mar  8 22:54:37 email kernel: [53025.709899] Pid: 14887, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:37 email kernel: [53025.709991] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:37 email kernel: [53025.710137] RSP: 0018:ffff8801cdd99c18  EFLAGS: 00010296
Mar  8 22:54:37 email kernel: [53025.710189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000122299
Mar  8 22:54:37 email kernel: [53025.710189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:37 email kernel: [53025.710189] RBP: ffff8801cdd99c18 R08: 0000000000000000 R09: ffff8801cdd99a68
Mar  8 22:54:37 email kernel: [53025.710189] R10: ffff88024dd99ab7 R11: ffff8801cdd99978 R12: 0000000000000002
Mar  8 22:54:37 email kernel: [53025.710189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801cdd99cf8
Mar  8 22:54:37 email kernel: [53025.710189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:37 email kernel: [53025.710189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:37 email kernel: [53025.710189] CR2: 00007f0b5ceeb000 CR3: 00000001ce456000 CR4: 00000000000006e0
Mar  8 22:54:37 email kernel: [53025.710189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:37 email kernel: [53025.710189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:37 email kernel: [53025.710189] Process xfs_fsr (pid: 14887, threadinfo ffff8801cdd98000, task ffff8802c07b0d80)
Mar  8 22:54:37 email kernel: [53025.710189] Stack:
Mar  8 22:54:37 email kernel: [53025.710189]  ffff8801cdd99ce8 ffffffff8118af48 ffff8801cdd99c58 ffffffff810d46cb
Mar  8 22:54:37 email kernel: [53025.710189] <0> ffff8801cdd99dc8 ffff88032f6daa80 ffff8801cdd99dc8 ffff88032c858190
Mar  8 22:54:37 email kernel: [53025.710189] <0> ffffffff811cf73e ffff8801c6f1e000 ffff8801cdd99cb8 0000000000000018
Mar  8 22:54:37 email kernel: [53025.710189] Call Trace:
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:37 email kernel: [53025.710189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:37 email kernel: [53025.710189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:37 email kernel: [53025.710189]  RSP <ffff8801cdd99c18>
Mar  8 22:54:37 email kernel: [53025.716551] ---[ end trace c02651fded7c4a2d ]---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Not an oops - it's an assertion fail which causes a deliberate trap for debugging.

That's not what I expected to see. Are you running out of space on your filesystem?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

OK, but is it normal that the load is >=2.0 after that (while the processor is still mostly idle).

Not even close:

# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             377G  192G  185G  51% /var

cheers,
Michael

PS: Sorry for the private Mail Dave, the "reply-all-forgetness" bit me ;-)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Those are the assertion failures i saw during yesterday's run. The process xfs_fsr hung on inode 536898141 and i couldn't kill it, even with "kill -9". I had to reboot to get everything back to normal.

cheers,
Michael

Mar  9 22:32:43 email kernel: [57269.991906] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:32:43 email kernel: [57269.991958] ------------[ cut here ]------------
Mar  9 22:32:43 email kernel: [57269.992034] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:32:43 email kernel: [57269.992106] invalid opcode: 0000 [#1] SMP 
Mar  9 22:32:43 email kernel: [57269.992273] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:32:43 email kernel: [57269.992364] CPU 2 
Mar  9 22:32:43 email kernel: [57269.992481] Pid: 32055, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:32:43 email kernel: [57269.998789] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:32:43 email kernel: [57269.998789] RSP: 0018:ffff8800614a7c18  EFLAGS: 00010296
Mar  9 22:32:43 email kernel: [57269.998789] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000000f093
Mar  9 22:32:43 email kernel: [57269.998789] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:32:43 email kernel: [57269.998789] RBP: ffff8800614a7c18 R08: 0000000000000002 R09: ffff8800614a7a68
Mar  9 22:32:43 email kernel: [57269.998789] R10: 0000000000000000 R11: 0000000000018600 R12: 0000000000000002
Mar  9 22:32:43 email kernel: [57269.998789] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8800614a7cf8
Mar  9 22:32:43 email kernel: [57269.998789] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:32:43 email kernel: [57269.998789] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:32:43 email kernel: [57269.998789] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:32:43 email kernel: [57269.998789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:32:43 email kernel: [57269.998789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:32:43 email kernel: [57269.998789] Process xfs_fsr (pid: 32055, threadinfo ffff8800614a6000, task ffff88032e0006c0)
Mar  9 22:32:43 email kernel: [57269.998789] Stack:
Mar  9 22:32:43 email kernel: [57269.998789]  ffff8800614a7ce8 ffffffff8118af48 ffff8800614a7c58 ffffffff810d46cb
Mar  9 22:32:43 email kernel: [57269.998789] <0> ffff8800614a7dc8 ffff88032c87d240 ffff8800614a7dc8 ffff88032c890190
Mar  9 22:32:43 email kernel: [57269.998789] <0> ffffffff811cf73e ffff88032dc96000 ffff8800614a7cb8 0000000000000018
Mar  9 22:32:43 email kernel: [57269.998789] Call Trace:
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:32:43 email kernel: [57269.998789] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:32:43 email kernel: [57269.998789] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:32:43 email kernel: [57269.998789]  RSP <ffff8800614a7c18>
Mar  9 22:32:43 email kernel: [57269.999136] ---[ end trace ee5cddd92301c840 ]---
Mar  9 22:33:14 email kernel: [57301.252722] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:33:14 email kernel: [57301.252868] ------------[ cut here ]------------
Mar  9 22:33:14 email kernel: [57301.252939] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:33:14 email kernel: [57301.253012] invalid opcode: 0000 [#2] SMP 
Mar  9 22:33:14 email kernel: [57301.253178] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:33:14 email kernel: [57301.253190] CPU 3 
Mar  9 22:33:14 email kernel: [57301.253190] Pid: 32080, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:33:14 email kernel: [57301.253190] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:14 email kernel: [57301.253190] RSP: 0018:ffff880285ea1c18  EFLAGS: 00010296
Mar  9 22:33:14 email kernel: [57301.253190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000000fcab
Mar  9 22:33:14 email kernel: [57301.253190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:33:14 email kernel: [57301.253190] RBP: ffff880285ea1c18 R08: 0000000000000000 R09: ffff880285ea1a68
Mar  9 22:33:14 email kernel: [57301.253190] R10: ffff880305ea1ab7 R11: ffff880285ea1978 R12: 0000000000000002
Mar  9 22:33:14 email kernel: [57301.253190] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880285ea1cf8
Mar  9 22:33:14 email kernel: [57301.253190] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:33:14 email kernel: [57301.253190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:33:14 email kernel: [57301.253190] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:33:14 email kernel: [57301.253190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:33:14 email kernel: [57301.253190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:33:14 email kernel: [57301.253190] Process xfs_fsr (pid: 32080, threadinfo ffff880285ea0000, task ffff88032e002880)
Mar  9 22:33:14 email kernel: [57301.253190] Stack:
Mar  9 22:33:14 email kernel: [57301.253190]  ffff880285ea1ce8 ffffffff8118af48 ffff880285ea1c58 ffffffff810d46cb
Mar  9 22:33:14 email kernel: [57301.253190] <0> ffff880285ea1dc8 ffff88032c87d240 ffff880285ea1dc8 ffff88032c890190
Mar  9 22:33:14 email kernel: [57301.253190] <0> ffffffff811cf73e ffff88032e934000 ffff880285ea1cb8 0000000000000018
Mar  9 22:33:14 email kernel: [57301.253190] Call Trace:
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:33:14 email kernel: [57301.253190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:33:14 email kernel: [57301.253190] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:14 email kernel: [57301.253190]  RSP <ffff880285ea1c18>
Mar  9 22:33:14 email kernel: [57301.260105] ---[ end trace ee5cddd92301c841 ]---
Mar  9 22:33:42 email kernel: [57329.070305] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:33:42 email kernel: [57329.070450] ------------[ cut here ]------------
Mar  9 22:33:42 email kernel: [57329.070522] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:33:42 email kernel: [57329.070594] invalid opcode: 0000 [#3] SMP 
Mar  9 22:33:42 email kernel: [57329.070760] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:33:42 email kernel: [57329.070851] CPU 3 
Mar  9 22:33:42 email kernel: [57329.070967] Pid: 32098, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:33:42 email kernel: [57329.071059] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:42 email kernel: [57329.071189] RSP: 0018:ffff8802dc321c18  EFLAGS: 00010296
Mar  9 22:33:42 email kernel: [57329.071189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 00000000000108cc
Mar  9 22:33:42 email kernel: [57329.071189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:33:42 email kernel: [57329.071189] RBP: ffff8802dc321c18 R08: 0000000000000000 R09: ffff8802dc321a68
Mar  9 22:33:42 email kernel: [57329.071189] R10: ffff88035c321ab7 R11: ffff8802dc321978 R12: 0000000000000002
Mar  9 22:33:42 email kernel: [57329.071189] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802dc321cf8
Mar  9 22:33:42 email kernel: [57329.071189] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:33:42 email kernel: [57329.071189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:33:42 email kernel: [57329.071189] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:33:42 email kernel: [57329.071189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:33:42 email kernel: [57329.071189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:33:42 email kernel: [57329.071189] Process xfs_fsr (pid: 32098, threadinfo ffff8802dc320000, task ffff88031eb8de80)
Mar  9 22:33:42 email kernel: [57329.071189] Stack:
Mar  9 22:33:42 email kernel: [57329.071189]  ffff8802dc321ce8 ffffffff8118af48 ffff8802dc321c58 ffffffff810d46cb
Mar  9 22:33:42 email kernel: [57329.071189] <0> ffff8802dc321dc8 ffff88032c87d240 ffff8802dc321dc8 ffff88032c890190
Mar  9 22:33:42 email kernel: [57329.071189] <0> ffffffff811cf73e ffff88032e932000 ffff8802dc321cb8 0000000000000018
Mar  9 22:33:42 email kernel: [57329.071189] Call Trace:
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:33:42 email kernel: [57329.071189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:33:42 email kernel: [57329.071189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:42 email kernel: [57329.071189]  RSP <ffff8802dc321c18>
Mar  9 22:33:42 email kernel: [57329.077632] ---[ end trace ee5cddd92301c842 ]---
Mar  9 22:34:10 email kernel: [57357.319260] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:34:10 email kernel: [57357.319405] ------------[ cut here ]------------
Mar  9 22:34:10 email kernel: [57357.319476] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:34:10 email kernel: [57357.319549] invalid opcode: 0000 [#4] SMP 
Mar  9 22:34:10 email kernel: [57357.319714] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:34:10 email kernel: [57357.319805] CPU 2 
Mar  9 22:34:10 email kernel: [57357.319922] Pid: 32107, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:34:10 email kernel: [57357.320014] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:10 email kernel: [57357.320127] RSP: 0018:ffff8802ed825c18  EFLAGS: 00010296
Mar  9 22:34:10 email kernel: [57357.320127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 00000000000114ed
Mar  9 22:34:10 email kernel: [57357.320127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:34:10 email kernel: [57357.320127] RBP: ffff8802ed825c18 R08: 0000000000000000 R09: ffff8802ed825a68
Mar  9 22:34:10 email kernel: [57357.320127] R10: ffff88036d825ab7 R11: ffff8802ed825978 R12: 0000000000000002
Mar  9 22:34:10 email kernel: [57357.320127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802ed825cf8
Mar  9 22:34:10 email kernel: [57357.320127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:34:10 email kernel: [57357.320127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:34:10 email kernel: [57357.320127] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:34:10 email kernel: [57357.320127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:34:10 email kernel: [57357.320127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:34:10 email kernel: [57357.320127] Process xfs_fsr (pid: 32107, threadinfo ffff8802ed824000, task ffff88031eb8ec00)
Mar  9 22:34:10 email kernel: [57357.320127] Stack:
Mar  9 22:34:10 email kernel: [57357.320127]  ffff8802ed825ce8 ffffffff8118af48 ffff8802ed825c58 ffffffff810d46cb
Mar  9 22:34:10 email kernel: [57357.320127] <0> ffff8802ed825dc8 ffff88032c87d240 ffff8802ed825dc8 ffff88032c890190
Mar  9 22:34:10 email kernel: [57357.320127] <0> ffffffff811cf73e ffff88032dc94000 ffff8802ed825cb8 0000000000000018
Mar  9 22:34:10 email kernel: [57357.320127] Call Trace:
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:34:10 email kernel: [57357.320127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:34:10 email kernel: [57357.320127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:10 email kernel: [57357.320127]  RSP <ffff8802ed825c18>
Mar  9 22:34:10 email kernel: [57357.332994] ---[ end trace ee5cddd92301c843 ]---
Mar  9 22:34:38 email kernel: [57385.263366] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:34:38 email kernel: [57385.263511] ------------[ cut here ]------------
Mar  9 22:34:38 email kernel: [57385.263583] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:34:38 email kernel: [57385.263656] invalid opcode: 0000 [#5] SMP 
Mar  9 22:34:38 email kernel: [57385.263821] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:34:38 email kernel: [57385.263912] CPU 2 
Mar  9 22:34:38 email kernel: [57385.264029] Pid: 32115, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:34:38 email kernel: [57385.264121] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:38 email kernel: [57385.264127] RSP: 0018:ffff880264af1c18  EFLAGS: 00010296
Mar  9 22:34:38 email kernel: [57385.264127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000001210e
Mar  9 22:34:38 email kernel: [57385.264127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:34:38 email kernel: [57385.264127] RBP: ffff880264af1c18 R08: 0000000000000000 R09: ffff880264af1a68
Mar  9 22:34:38 email kernel: [57385.264127] R10: ffff8802e4af1ab7 R11: ffff880264af1978 R12: 0000000000000002
Mar  9 22:34:38 email kernel: [57385.264127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880264af1cf8
Mar  9 22:34:38 email kernel: [57385.264127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:34:38 email kernel: [57385.264127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:34:38 email kernel: [57385.264127] CR2: 00007fe9c26ff000 CR3: 00000001e74f6000 CR4: 00000000000006e0
Mar  9 22:34:38 email kernel: [57385.264127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:34:38 email kernel: [57385.264127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:34:38 email kernel: [57385.264127] Process xfs_fsr (pid: 32115, threadinfo ffff880264af0000, task ffff88031eb8a1c0)
Mar  9 22:34:38 email kernel: [57385.264127] Stack:
Mar  9 22:34:38 email kernel: [57385.264127]  ffff880264af1ce8 ffffffff8118af48 ffff880264af1c58 ffffffff810d46cb
Mar  9 22:34:38 email kernel: [57385.264127] <0> ffff880264af1dc8 ffff88032c87d240 ffff880264af1dc8 ffff88032c890190
Mar  9 22:34:38 email kernel: [57385.264127] <0> ffffffff811cf73e ffff88032dc92000 ffff880264af1cb8 0000000000000018
Mar  9 22:34:38 email kernel: [57385.264127] Call Trace:
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:34:38 email kernel: [57385.264127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:34:38 email kernel: [57385.264127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:38 email kernel: [57385.264127]  RSP <ffff880264af1c18>
Mar  9 22:34:38 email kernel: [57385.270689] ---[ end trace ee5cddd92301c844 ]---
Mar  9 22:35:05 email kernel: [57412.310400] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:05 email kernel: [57412.310547] ------------[ cut here ]------------
Mar  9 22:35:05 email kernel: [57412.310618] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:05 email kernel: [57412.310691] invalid opcode: 0000 [#6] SMP 
Mar  9 22:35:05 email kernel: [57412.310856] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:05 email kernel: [57412.310947] CPU 2 
Mar  9 22:35:05 email kernel: [57412.311064] Pid: 32120, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:05 email kernel: [57412.311127] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:05 email kernel: [57412.311127] RSP: 0018:ffff880289b33c18  EFLAGS: 00010296
Mar  9 22:35:05 email kernel: [57412.311127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000012d2f
Mar  9 22:35:05 email kernel: [57412.311127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:05 email kernel: [57412.311127] RBP: ffff880289b33c18 R08: 0000000000000000 R09: ffff880289b33a68
Mar  9 22:35:05 email kernel: [57412.311127] R10: ffff880309b33ab7 R11: ffff880289b33978 R12: 0000000000000002
Mar  9 22:35:05 email kernel: [57412.311127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880289b33cf8
Mar  9 22:35:05 email kernel: [57412.311127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:35:05 email kernel: [57412.311127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:05 email kernel: [57412.311127] CR2: 00007fe9c26ff000 CR3: 00000001d4e56000 CR4: 00000000000006e0
Mar  9 22:35:05 email kernel: [57412.311127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:05 email kernel: [57412.311127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:05 email kernel: [57412.311127] Process xfs_fsr (pid: 32120, threadinfo ffff880289b32000, task ffff88030cbe6540)
Mar  9 22:35:05 email kernel: [57412.311127] Stack:
Mar  9 22:35:05 email kernel: [57412.311127]  ffff880289b33ce8 ffffffff8118af48 ffff880289b33c58 ffffffff810d46cb
Mar  9 22:35:05 email kernel: [57412.311127] <0> ffff880289b33dc8 ffff88032c87d240 ffff880289b33dc8 ffff88032c890190
Mar  9 22:35:05 email kernel: [57412.311127] <0> ffffffff811cf73e ffff88032d44a000 ffff880289b33cb8 0000000000000018
Mar  9 22:35:05 email kernel: [57412.311127] Call Trace:
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:05 email kernel: [57412.311127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:05 email kernel: [57412.311127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:05 email kernel: [57412.311127]  RSP <ffff880289b33c18>
Mar  9 22:35:05 email kernel: [57412.317729] ---[ end trace ee5cddd92301c845 ]---
Mar  9 22:35:30 email kernel: [57437.288969] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:30 email kernel: [57437.289114] ------------[ cut here ]------------
Mar  9 22:35:30 email kernel: [57437.289186] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:30 email kernel: [57437.289190] invalid opcode: 0000 [#7] SMP 
Mar  9 22:35:30 email kernel: [57437.289190] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:30 email kernel: [57437.289190] CPU 3 
Mar  9 22:35:30 email kernel: [57437.289190] Pid: 32129, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:30 email kernel: [57437.289190] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:30 email kernel: [57437.289190] RSP: 0018:ffff8801e41c5c18  EFLAGS: 00010296
Mar  9 22:35:30 email kernel: [57437.289190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000013950
Mar  9 22:35:30 email kernel: [57437.289190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:30 email kernel: [57437.289190] RBP: ffff8801e41c5c18 R08: 0000000000000000 R09: ffff8801e41c5a68
Mar  9 22:35:30 email kernel: [57437.289190] R10: ffff8802641c5ab7 R11: ffff8801e41c5978 R12: 0000000000000002
Mar  9 22:35:30 email kernel: [57437.289190] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8801e41c5cf8
Mar  9 22:35:30 email kernel: [57437.289190] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:35:30 email kernel: [57437.289190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:30 email kernel: [57437.289190] CR2: 00007fe9c26ff000 CR3: 0000000115bfc000 CR4: 00000000000006e0
Mar  9 22:35:30 email kernel: [57437.289190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:30 email kernel: [57437.289190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:30 email kernel: [57437.289190] Process xfs_fsr (pid: 32129, threadinfo ffff8801e41c4000, task ffff88030cbe4a40)
Mar  9 22:35:30 email kernel: [57437.289190] Stack:
Mar  9 22:35:30 email kernel: [57437.289190]  ffff8801e41c5ce8 ffffffff8118af48 ffff8801e41c5c58 ffffffff810d46cb
Mar  9 22:35:30 email kernel: [57437.289190] <0> ffff8801e41c5dc8 ffff88032c87d240 ffff8801e41c5dc8 ffff88032c890190
Mar  9 22:35:30 email kernel: [57437.289190] <0> ffffffff811cf73e ffff880289804000 ffff8801e41c5cb8 0000000000000018
Mar  9 22:35:30 email kernel: [57437.289190] Call Trace:
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:30 email kernel: [57437.289190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:30 email kernel: [57437.289190] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:30 email kernel: [57437.289190]  RSP <ffff8801e41c5c18>
Mar  9 22:35:30 email kernel: [57437.302717] ---[ end trace ee5cddd92301c846 ]---
Mar  9 22:35:52 email kernel: [57458.749511] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:52 email kernel: [57458.749657] ------------[ cut here ]------------
Mar  9 22:35:52 email kernel: [57458.749729] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:52 email kernel: [57458.749801] invalid opcode: 0000 [#8] SMP 
Mar  9 22:35:52 email kernel: [57458.749967] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:52 email kernel: [57458.750058] CPU 2 
Mar  9 22:35:52 email kernel: [57458.750128] Pid: 32140, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:52 email kernel: [57458.750128] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:52 email kernel: [57458.750128] RSP: 0018:ffff8802676f5c18  EFLAGS: 00010296
Mar  9 22:35:52 email kernel: [57458.750128] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000014571
Mar  9 22:35:52 email kernel: [57458.750128] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:52 email kernel: [57458.750128] RBP: ffff8802676f5c18 R08: 0000000000000000 R09: ffff8802676f5a68
Mar  9 22:35:52 email kernel: [57458.750128] R10: ffff8802e76f5ab7 R11: ffff8802676f5978 R12: 0000000000000002
Mar  9 22:35:52 email kernel: [57458.750128] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802676f5cf8
Mar  9 22:35:52 email kernel: [57458.750128] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:35:52 email kernel: [57458.750128] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:52 email kernel: [57458.750128] CR2: 00007fe9c26ff000 CR3: 0000000115bfc000 CR4: 00000000000006e0
Mar  9 22:35:52 email kernel: [57458.750128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:52 email kernel: [57458.750128] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:52 email kernel: [57458.750128] Process xfs_fsr (pid: 32140, threadinfo ffff8802676f4000, task ffff88031eb8a880)
Mar  9 22:35:52 email kernel: [57458.750128] Stack:
Mar  9 22:35:52 email kernel: [57458.750128]  ffff8802676f5ce8 ffffffff8118af48 ffff8802676f5c58 ffffffff810d46cb
Mar  9 22:35:52 email kernel: [57458.750128] <0> ffff8802676f5dc8 ffff88032c87d240 ffff8802676f5dc8 ffff88032c890190
Mar  9 22:35:52 email kernel: [57458.750128] <0> ffffffff811cf73e ffff88032d44c000 ffff8802676f5cb8 0000000000000018
Mar  9 22:35:52 email kernel: [57458.750128] Call Trace:
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:52 email kernel: [57458.750128] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:52 email kernel: [57458.750128] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:52 email kernel: [57458.750128]  RSP <ffff8802676f5c18>
Mar  9 22:35:52 email kernel: [57458.756836] ---[ end trace ee5cddd92301c847 ]---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Michael - have you got any idea what the files are that are hitting this? This failure is implying that the inode is still dirty after syncing all the data. Is something trying to modify it while XFS is trying to map it?

We've seen this occasionally in the past, but we've never been able to reproduce it with any reliability. Hence any information you can extract would certainly help us here.

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Dave!

Yes, as far as i can tell it's always a file that some process is currently modifying. It happens ofter with some file unter /var/log which syslog is currently modifying. I tried setting the "no-defrag" flag via xfs_io's chattr on all log files but that didn't seem to help. It seems that cyrus imapd is triggering this problem far more likely than any other program. Some examples of files where it usually hangs:
/var/spool/imap/x/user/xxxx/cyrus.cache (lsof -> cyrus)
/var/imap/db/log.xxxxxxx (lsof -> cyrus)
/var/log/xxx.log (lsof -> syslog)

I've also seen this happen occasionally before, but recently it started to happen on about every second run of xfs_fsr. Because this is a production system i've ceased using xfs_fsr for the moment. I've not been able to reproduce it on others systems yet.

cheers,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

So what's interesting is that cyrus uses mmapp access to files, which might be an indicator that we have problems with excluding fsr on mmaped files.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

From what i saw by experimenting xfs_fsr doesn't skip mmapped files (while imho it should). Another cyrus file it hung on recently was:
/var/imap/user/x/xxxxxxxx.seen: Cyrus skiplist DB

I will be happy to test out any patches.

hth,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Ah, yeah.

->page_mkwrite executes without the inode iolock held, so we can't lock it out from creating new delalloc pages by holding the iolock like the bmap code does.

I don't think we're allowed to take the iolock in ->page_mkwrite, so effectively that leaves us with the situation where we can't do an atomic flush and map in the bmap code.

Christoph, I guess that means we need to make the bmap code handle/ignore delalloc extents rather than assume they never occur after the flush. What do you think?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

The current swapext code is supposed to skip any file that is mapped into userspace via mmap. So if you get a file that is actually mmaped something is wrong with that check. From a quick look the only problem I see is that we don't take the iolock in ->mmap so we could add a new mapping after the check, but I'd be surprise if that is what Michael is seeing.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

The swapext mmap check occurs a long time after we actually get the extents from the file, so while the swapext will fail that check, we're not getting to it. i.e. we're assert failing during the setup for the data move before we do the swapext. Perhaps we should make xfs_getbmap() return EBUSY for mmap()d files like swapext does?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Sorry I only found this thread now after looking for xfs_fsr related threads, and I don't follow the list regularly currently.

I've had crashes once with xfs_fsr, only to find that my filesystem was a bit broken (which lead to an improved xfs_repair, btw).

Michael, did you try to xfs_repair once to check if you got errors?

mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Michael!

Yes, as already said before i ran xfs_repair (version 3.1.1) on the fs and it reported no errors. For the moment i had to disable xfs_fsr on that server since it is a production machine.

with kind regards,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs