Shutdown preventing umount
Hi,
I had a cable problem on a USB connected XFS file system, triggering some IO errors, and the result was that any access to the mount point resulted in EIO. This prevented unmounting the file system to recover from the problem.
I also tried xfs_io with shutdown -f, but it had the same problem because xfs_io couldn't access anything on the file system.
How is that supposed to work? Having to reboot just to recover from IO errors doesn't seem to be very available.
I don't think shutdown should prevent unmounting.
From googling I found some old RHEL bugzilla that such a problem was fixed in some RHEL release. Is that a regression?
This was on a 5.4.10 kernel.
I got lots of:
XFS (...): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x4a620288 len 8 error 5
Then some
XFS (...): writeback error on sector 7372099184
And finally:
XFS (...): log I/O error -5
XFS (...): xfs_do_force_shutdown(0x2)
called from line 1250 of file fs/xfs/xfs_log.c. Return address = 00000000f7956130
XFS (...): Log I/O Error Detected.
Shutting down filesystem
XFS (...): Please unmount the filesystem
and rectify the problem(s)
(very funny XFS!)
XFS (...): log I/O error -5
scsi 7:0:0:0: rejecting I/O to dead device
-Andi
Quoted 7 lines · Andi Kleen
Hi,
I had a cable problem on a USB connected XFS file system, triggering some IO errors, and the result was that any access to the mount point resulted in EIO. This prevented unmounting the file system to recover from the problem.
Full dmesg output, please.
Quoted 2 lines
I also tried
xfs_iowith shutdown -f, but it had the same problem becausexfs_iocouldn't access anything on the file system.
Because the IO errors had already shut the filesystem down, as per the dmesg output you quoted below.
Quoted 4 lines
How is that supposed to work? Having to reboot just to recover from IO errors doesn't seem to be very available.
I don't think shutdown should prevent unmounting.
It doesn't. Something was leaked or not cleaned up properly, preventing the filesytem from being unmounted. You know, a bug...
Quoted 2 lines
From googling I found some old RHEL bugzilla that such a problem was fixed in some RHEL release. Is that a regression?
RHEL has an upstream first policy, so whatever bug fix you find in a RHEL kernel is already in the upstream kernels.
Quoted 22 lines
This was on a 5.4.10 kernel.
I got lots of:
XFS (...): metadata I/O error in "
xfs_trans_read_buf_map" at daddr0x4a620288len 8 error 5Then some
XFS (...): writeback error on sector 7372099184
And finally:
XFS (...): log I/O error -5
XFS (...):xfs_do_force_shutdown(0x2)
called from line 1250 of filefs/xfs/xfs_log.c. Return address =00000000f7956130
XFS (...): Log I/O Error Detected.
Shutting down filesystem
XFS (...): Please unmount the filesystem
and rectify the problem(s)
Quoted 5 lines
(very funny XFS!)
XFS (...): log I/O error -5
scsi 7:0:0:0: rejecting I/O to dead device
Where is unmount stuck? 'echo w > /proc/sysrq-trigger' output if it is hung, 'echo l > /proc/sysrq-trigger' if it is spinning, please?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 10 lines · Dave Chinner
Quoted 7 lines · Andi Kleen
Hi,
I had a cable problem on a USB connected XFS file system, triggering some IO errors, and the result was that any access to the mount point resulted in EIO. This prevented unmounting the file system to recover from the problem.
Full dmesg output, please.
http://www.firstfloor.org/~andi/dmesg-xfs-umount
Quoted 7 lines
Quoted 4 lines
XFS (...): log I/O error -5
scsi 7:0:0:0: rejecting I/O to dead device
Where is unmount stuck? 'echo w >
/proc/sysrq-trigger' output if it is hung, 'echo l >/proc/sysrq-trigger' if it is spinning, please?
It's not stuck. It always errored out with EIO.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Dave Chinner
Quoted 3 lines · Andi Kleen
Hi,
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I had a cable problem on a USB connected XFS file system, triggering some IO errors, and the result was that any access to the mount point resulted in EIO. This prevented unmounting the file system to recover from the problem.
Full dmesg output, please.
So /dev/sdd went away, XFS shutdown eventually. There's no indication that an XFS unmount has started, because the first thing XFS does in it's ->put_super method is emit this to the log:
XFS (dm-4): Unmounting Filesystem
and that is missing. Hence the VFS unmount path has not reached as far as XFS before it has errorred out. THis is confirmed by....
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
XFS (...): log I/O error -5
scsi 7:0:0:0: rejecting I/O to dead device
Where is unmount stuck? 'echo w >
/proc/sysrq-trigger' output if it is hung, 'echo l >/proc/sysrq-trigger' if it is spinning, please?It's not stuck. It always errored out with EIO.
... the fact that filesystems cannot return errors from unmount proceedings as generic_shutdown_super() is a void function. Hence where this EIO is coming from is not obvious - it isn't from XFS failing to unmount the filesystem as it's not gettting that far into the unmount path.
You're going to need to strace umount to find what syscall is failing, then probably use tracepoints or some one kernel introspection tool to work out where the EIO is coming from...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]