XFS Failed to recover EFIs - XFS_WANT_CORRUPTED_GOTO at line 1602 of xfs_alloc.c
Hi,
A virtual server of mine stopped working properly yesterday because one partition became corrupted (or corruption has been stumbled over).
Restarting the system any attempt to mount that partition (without -o norecovery,ro) results in the following trace (transcribed):
XFS (sda5): Mounting Filesystem
XFS (sda5): Starting recovery (logdev: internal)
XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1602 of file
/var/cache/kernel/linux-git/fs/xfs/xfs_alloc.c. Caller0xffffffff8116d926
CPU: 0 PID: 606 Commm: mount Not tainted 3.13.0-hetzner #1 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 000000000002eb84 ffff88001dc53ab8 ffffffff813ca339 ffff88001dc53ad8 ffffffff81156d4a ffffffff8116d926 00000000000002a8 ffff88001dc53b68 ffffffff8116b8dd ffff88001dd7ccc0 0000000000000000 0000000000000001 Call Trace: [<ffffffff813ca339>] dump_stack+0x19/0x1b [<ffffffff81156d4a>] xfs_error_report+0x3a0x40 [<ffffffff8116d926>] ? xfs_free_extent+0xd6/0x120 [<ffffffff8116b8dd>] xfs_free_ag_extent+0x48d/0x5c0 [<ffffffff8116d926>] xfs_free_extent+0xd6/0x120 [<ffffffff810d5fa4>] ? kmem_cache_alloc+0xa4/0xb0 [<ffffffff8119c390>] xlog_recover_process_efi+0x170/0x1b0 [<ffffffff81074709>] ? wake_up_bit+0x29/0x40 [<ffffffff8119d106>] xlog_recover_process_efis.isra.27+0x46/0x80 [<ffffffff811a17c5>] xlog_recover_finish+0x2c/0x50 [<ffffffff811a5c4c>] xfs_log_mount_finish+0x2c/0x50 [<ffffffff811958ee>] ? xfs_iunlock+0x6e/0x90 [<ffffffff81164733>] xfs_mountfs+0x473/0x690 [<ffffffff81167072>] xfs_fs_fill_super+0x292/0x310 [<ffffffff810e7a61>] mount_bdev+0x191/0x1d0 [<ffffffff811e337c>] ? ida_get_new_above+0x21c/0x290 [<ffffffff81166de0>] ? xfs_parseargs+0xc10/0xc10 [<ffffffff81165310>] xfs_fs_mount+0x10/0x20 [<ffffffff810e7cab>] mount_fs+0x1b/0xd0 [<ffffffff811001ad>] vfs_kern_mount+0x6d/0x100 [<ffffffff811019bb>] do_mount+0x1fb/0x9d0 [<ffffffff810b3b43>] ? strndup_user+0x53/0x70 [<ffffffff81102469>] SyS_mount+0x89/0xd0 [<ffffffff831ce4b7>] system_call_fastpath+0x16/0x1b
XFS (sda5): Failed to recover EFIs
XFS (sda5): log mount finish failed
After that the mount process remains in D state and any attempt to xfs_repair that fileysystem blocks (reboot needed to do anything).
Is that expected or should the mount either completely fail, returning proper error to mount and leave system in a state as if the mount had never been attempted (except for the log messages)?
Quoted 1 line
From the cause of this, I guess it's some left-over of "unclean"
live migration of the KVM guest this system is running on some longer time ago. After live migration some processes started dying weird deaths. Rebooting the system worked fine by the time though.
The only major load on that system (not so heavy, about 10-20 IO-ops per second on average, mostly writes) is updating RRD files and running a slave MySQL (InnoDB) database.
I recovered the filesystem with xfs_repair -L /dev/sda5 though the InnoDB state remaining is rather broken.xfs_repair reported only claimed free space issues (I didn't save its output).
Bruno
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 43 lines · Bruno Prémont
Hi,
A virtual server of mine stopped working properly yesterday because one partition became corrupted (or corruption has been stumbled over).
Restarting the system any attempt to mount that partition (without -o norecovery,ro) results in the following trace (transcribed):
XFS (sda5): Mounting Filesystem
XFS (sda5): Starting recovery (logdev: internal)
XFS: Internal errorXFS_WANT_CORRUPTED_GOTOat line 1602 of file
/var/cache/kernel/linux-git/fs/xfs/xfs_alloc.c. Caller0xffffffff8116d926CPU: 0 PID: 606 Commm: mount Not tainted 3.13.0-hetzner #1 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 000000000002eb84 ffff88001dc53ab8 ffffffff813ca339 ffff88001dc53ad8 ffffffff81156d4a ffffffff8116d926 00000000000002a8 ffff88001dc53b68 ffffffff8116b8dd ffff88001dd7ccc0 0000000000000000 0000000000000001 Call Trace: [<ffffffff813ca339>] dump_stack+0x19/0x1b [<ffffffff81156d4a>] xfs_error_report+0x3a0x40 [<ffffffff8116d926>] ? xfs_free_extent+0xd6/0x120 [<ffffffff8116b8dd>] xfs_free_ag_extent+0x48d/0x5c0 [<ffffffff8116d926>] xfs_free_extent+0xd6/0x120 [<ffffffff810d5fa4>] ? kmem_cache_alloc+0xa4/0xb0 [<ffffffff8119c390>] xlog_recover_process_efi+0x170/0x1b0 [<ffffffff81074709>] ? wake_up_bit+0x29/0x40 [<ffffffff8119d106>] xlog_recover_process_efis.isra.27+0x46/0x80 [<ffffffff811a17c5>] xlog_recover_finish+0x2c/0x50 [<ffffffff811a5c4c>] xfs_log_mount_finish+0x2c/0x50 [<ffffffff811958ee>] ? xfs_iunlock+0x6e/0x90 [<ffffffff81164733>] xfs_mountfs+0x473/0x690 [<ffffffff81167072>] xfs_fs_fill_super+0x292/0x310 [<ffffffff810e7a61>] mount_bdev+0x191/0x1d0 [<ffffffff811e337c>] ? ida_get_new_above+0x21c/0x290 [<ffffffff81166de0>] ? xfs_parseargs+0xc10/0xc10 [<ffffffff81165310>] xfs_fs_mount+0x10/0x20 [<ffffffff810e7cab>] mount_fs+0x1b/0xd0 [<ffffffff811001ad>] vfs_kern_mount+0x6d/0x100 [<ffffffff811019bb>] do_mount+0x1fb/0x9d0 [<ffffffff810b3b43>] ? strndup_user+0x53/0x70 [<ffffffff81102469>] SyS_mount+0x89/0xd0 [<ffffffff831ce4b7>] system_call_fastpath+0x16/0x1b
Quoted 2 lines
XFS (sda5): Failed to recover EFIs
XFS (sda5): log mount finish failed
curious on which version of Linux hit this problem?
Quoted 8 lines
After that the mount process remains in D state and any attempt to
xfs_repairthat fileysystem blocks (reboot needed to do anything).Is that expected or should the mount either completely fail, returning proper error to mount and leave system in a state as if the mount had never been attempted (except for the log messages)?
The xfs_ail_push_all_sync() is hanging because the EFI was not and will not be removed. There is a patch for this problem, but is waiting for a similar issue in xlog_cil_push() that would change the recovery patch.
Quoted 16 lines
Quoted 3 lines
From the cause of this, I guess it's some left-over of "unclean"
live migration of the KVM guest this system is running on some longer time ago. After live migration some processes started dying weird deaths. Rebooting the system worked fine by the time though.
The only major load on that system (not so heavy, about 10-20 IO-ops per second on average, mostly writes) is updating RRD files and running a slave MySQL (InnoDB) database.
I recovered the filesystem with
xfs_repair-L/dev/sda5though the InnoDB state remaining is rather broken.xfs_repairreported only claimed free space issues (I didn't save its output).
Quoted 6 lines
Bruno
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
--Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · On Fri, 21 Feb 2014 08:14:12 -0600 Mark Tinguely wrote:
Quoted 2 lines · Bruno Prémont
A virtual server of mine stopped working properly yesterday because one partition became corrupted (or corruption has been stumbled over).
The running kernel was 3.12.6.
I would have appreciated if the XFS filesystem had continued being accessible even if only in read-only mode instead of completely shutting down. That would have made it possible to gather more information and doing so more easily as well.
Quoted 40 lines
Quoted 37 lines
Restarting the system any attempt to mount that partition (without -o norecovery,ro) results in the following trace (transcribed):
XFS (sda5): Mounting Filesystem
XFS (sda5): Starting recovery (logdev: internal)
XFS: Internal errorXFS_WANT_CORRUPTED_GOTOat line 1602 of file
/var/cache/kernel/linux-git/fs/xfs/xfs_alloc.c. Caller0xffffffff8116d926CPU: 0 PID: 606 Commm: mount Not tainted 3.13.0-hetzner #1 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 000000000002eb84 ffff88001dc53ab8 ffffffff813ca339 ffff88001dc53ad8 ffffffff81156d4a ffffffff8116d926 00000000000002a8 ffff88001dc53b68 ffffffff8116b8dd ffff88001dd7ccc0 0000000000000000 0000000000000001 Call Trace: [<ffffffff813ca339>] dump_stack+0x19/0x1b [<ffffffff81156d4a>] xfs_error_report+0x3a0x40 [<ffffffff8116d926>] ? xfs_free_extent+0xd6/0x120 [<ffffffff8116b8dd>] xfs_free_ag_extent+0x48d/0x5c0 [<ffffffff8116d926>] xfs_free_extent+0xd6/0x120 [<ffffffff810d5fa4>] ? kmem_cache_alloc+0xa4/0xb0 [<ffffffff8119c390>] xlog_recover_process_efi+0x170/0x1b0 [<ffffffff81074709>] ? wake_up_bit+0x29/0x40 [<ffffffff8119d106>] xlog_recover_process_efis.isra.27+0x46/0x80 [<ffffffff811a17c5>] xlog_recover_finish+0x2c/0x50 [<ffffffff811a5c4c>] xfs_log_mount_finish+0x2c/0x50 [<ffffffff811958ee>] ? xfs_iunlock+0x6e/0x90 [<ffffffff81164733>] xfs_mountfs+0x473/0x690 [<ffffffff81167072>] xfs_fs_fill_super+0x292/0x310 [<ffffffff810e7a61>] mount_bdev+0x191/0x1d0 [<ffffffff811e337c>] ? ida_get_new_above+0x21c/0x290 [<ffffffff81166de0>] ? xfs_parseargs+0xc10/0xc10 [<ffffffff81165310>] xfs_fs_mount+0x10/0x20 [<ffffffff810e7cab>] mount_fs+0x1b/0xd0 [<ffffffff811001ad>] vfs_kern_mount+0x6d/0x100 [<ffffffff811019bb>] do_mount+0x1fb/0x9d0 [<ffffffff810b3b43>] ? strndup_user+0x53/0x70 [<ffffffff81102469>] SyS_mount+0x89/0xd0 [<ffffffff831ce4b7>] system_call_fastpath+0x16/0x1bQuoted 2 lines
XFS (sda5): Failed to recover EFIs
XFS (sda5): log mount finish failed
Quoted 1 line
curious on which version of Linux hit this problem?
The trace was produced by 3.13 kernel from kernel.org.
A reboot attempt with 3.12.6 showed a similar trace though I didn't record it.
Quoted 24 lines
Quoted 6 lines
After that the mount process remains in D state and any attempt to
xfs_repairthat fileysystem blocks (reboot needed to do anything).Is that expected or should the mount either completely fail, returning proper error to mount and leave system in a state as if the mount had never been attempted (except for the log messages)?
The
xfs_ail_push_all_sync()is hanging because the EFI was not and will not be removed. There is a patch for this problem, but is waiting for a similar issue inxlog_cil_push()that would change the recovery patch.Quoted 13 lines
Quoted 1 line
From the cause of this, I guess it's some left-over of "unclean"
live migration of the KVM guest this system is running on some longer time ago. After live migration some processes started dying weird deaths. Rebooting the system worked fine by the time though.
The only major load on that system (not so heavy, about 10-20 IO-ops per second on average, mostly writes) is updating RRD files and running a slave MySQL (InnoDB) database.
I recovered the filesystem with
xfs_repair-L/dev/sda5though the InnoDB state remaining is rather broken.xfs_repairreported only claimed free space issues (I didn't save its output).
Bruno
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Bruno Prémont
Quoted 3 lines · On Fri, 21 Feb 2014 08:14:12 -0600 Mark Tinguely wrote:
Quoted 2 lines · Bruno Prémont
A virtual server of mine stopped working properly yesterday because one partition became corrupted (or corruption has been stumbled over).
The running kernel was 3.12.6.
I would have appreciated if the XFS filesystem had continued being accessible even if only in read-only mode instead of completely shutting down. That would have made it possible to gather more information and doing so more easily as well.
Well it appears that the xlog_cil_push problem is not going to be fixed soon, so I will repost the patch for the log recovery portion.
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Restarting the system any attempt to mount that partition (without -o norecovery,ro) results in the following trace (transcribed):
XFS (sda5): Mounting Filesystem
XFS (sda5): Starting recovery (logdev: internal)
XFS: Internal errorXFS_WANT_CORRUPTED_GOTOat line 1602 of file
/var/cache/kernel/linux-git/fs/xfs/xfs_alloc.c. Caller0xffffffff8116d926CPU: 0 PID: 606 Commm: mount Not tainted 3.13.0-hetzner #1 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 000000000002eb84 ffff88001dc53ab8 ffffffff813ca339 ffff88001dc53ad8 ffffffff81156d4a ffffffff8116d926 00000000000002a8 ffff88001dc53b68 ffffffff8116b8dd ffff88001dd7ccc0 0000000000000000 0000000000000001 Call Trace: [<ffffffff813ca339>] dump_stack+0x19/0x1b [<ffffffff81156d4a>] xfs_error_report+0x3a0x40 [<ffffffff8116d926>] ? xfs_free_extent+0xd6/0x120 [<ffffffff8116b8dd>] xfs_free_ag_extent+0x48d/0x5c0 [<ffffffff8116d926>] xfs_free_extent+0xd6/0x120 [<ffffffff810d5fa4>] ? kmem_cache_alloc+0xa4/0xb0 [<ffffffff8119c390>] xlog_recover_process_efi+0x170/0x1b0 [<ffffffff81074709>] ? wake_up_bit+0x29/0x40 [<ffffffff8119d106>] xlog_recover_process_efis.isra.27+0x46/0x80 [<ffffffff811a17c5>] xlog_recover_finish+0x2c/0x50 [<ffffffff811a5c4c>] xfs_log_mount_finish+0x2c/0x50 [<ffffffff811958ee>] ? xfs_iunlock+0x6e/0x90 [<ffffffff81164733>] xfs_mountfs+0x473/0x690 [<ffffffff81167072>] xfs_fs_fill_super+0x292/0x310 [<ffffffff810e7a61>] mount_bdev+0x191/0x1d0 [<ffffffff811e337c>] ? ida_get_new_above+0x21c/0x290 [<ffffffff81166de0>] ? xfs_parseargs+0xc10/0xc10 [<ffffffff81165310>] xfs_fs_mount+0x10/0x20 [<ffffffff810e7cab>] mount_fs+0x1b/0xd0 [<ffffffff811001ad>] vfs_kern_mount+0x6d/0x100 [<ffffffff811019bb>] do_mount+0x1fb/0x9d0 [<ffffffff810b3b43>] ? strndup_user+0x53/0x70 [<ffffffff81102469>] SyS_mount+0x89/0xd0 [<ffffffff831ce4b7>] system_call_fastpath+0x16/0x1bXFS (sda5): Failed to recover EFIs
XFS (sda5): log mount finish failed
Quoted 6 lines
Quoted 1 line
curious on which version of Linux hit this problem?
The trace was produced by 3.13 kernel from kernel.org.
A reboot attempt with 3.12.6 showed a similar trace though I didn't record it.
The original problem is attempting to free a partial free extent,
I was curious because that problem has been flaring up recently internally and in the community.
Thank-you for the information.
--Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs