XFS Mount Hangs the Partition (on latest kernel + many old 2.6.x ones)
Hi all!
(Please CC me on replies)
I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "xfs_check" or "xfs_repair") to hang too. However, running xfs_check
or xfs_repair before the first mount (after a reboot) worked, and eventually resolved this problem.
I blogged about it (relatively incoherently) here:
http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply
It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.
Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).
After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that xfs_check and xfs_repair both hanged up as well. After a reboot, I tried running xfs_check right away on that partition and it worked. So I ran xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.
I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by xfs_check and xfs_repair to fail as well.
I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.
Regards,
Shlomi Fish
---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/
95% of the programmers consider 95% of the code they did not write, in the bottom 5%.
Quoted 1 line · Shlomi Fish
Hi all!
Hi there,
Quoted 10 lines
(Please CC me on replies)
I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "
xfs_check" or "xfs_repair") to hang too. However, runningxfs_check
orxfs_repairbefore the first mount (after a reboot) worked, and eventually resolved this problem.
Quoted 4 lines
I blogged about it (relatively incoherently) here:
http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply
Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.
Quoted 5 lines
It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.
Quoted 8 lines
Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).
After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that
xfs_checkandxfs_repairboth hanged up as well. After a reboot, I tried runningxfs_check
This was probably caused by the block device being held open exclusively by the stuck mount process.
Quoted 11 lines
right away on that partition and it worked. So I ran
xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by
xfs_checkandxfs_repairto fail as well.
Quoted 3 lines
I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.
Thats unlikely to help now - repair will have wiped your log clean, so all evidence of the problem will be gone.
cheers.
Signature · 1 line
Nathan
Quoted 4 lines · Nathan Scott
Quoted 1 line · Shlomi Fish
Hi all!
Hi there,
Hi Mr. Scott (and all)!
Quoted 16 lines
Quoted 11 lines
(Please CC me on replies)
I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "
xfs_check" or "xfs_repair") to hang too. However, runningxfs_check
orxfs_repairbefore the first mount (after a reboot) worked, and eventually resolved this problem.Quoted 4 lines
I blogged about it (relatively incoherently) here:
http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply
Quoted 6 lines
Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- sendxfs_infooutput for the filesystem in question;
- dump the log (xfs_logprint-C) and send it to us.
Sure. But in what order should I do all that?
Quoted 12 lines
Quoted 12 lines
It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.
Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).
Quoted 7 lines
Quoted 4 lines
After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that
xfs_checkandxfs_repairboth hanged up as well. After a reboot, I tried runningxfs_checkThis was probably caused by the block device being held open exclusively by the stuck mount process.
I see.
Quoted 14 lines
Quoted 14 lines
right away on that partition and it worked. So I ran
xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by
xfs_checkandxfs_repairto fail as well.
Quoted 6 lines
Quoted 3 lines
I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.
Thats unlikely to help now - repair will have wiped your log clean, so all evidence of the problem will be gone.
Yes. I thought one can try looking in the XFS mounting code for possible bugs.
Regards,
Shlomi Fish
---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/
95% of the programmers consider 95% of the code they did not write, in the bottom 5%.
Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.
Regards,
Shlomi Fish
Quoted 23 lines · Shlomi Fish
Quoted 4 lines · Nathan Scott
Quoted 1 line · Shlomi Fish
Hi all!
Hi there,
Hi Mr. Scott (and all)!
Quoted 15 lines
Quoted 14 lines
(Please CC me on replies)
I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "
xfs_check" or "xfs_repair") to hang too. However, runningxfs_check
orxfs_repairbefore the first mount (after a reboot) worked, and eventually resolved this problem.I blogged about it (relatively incoherently) here:
http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply
Quoted 8 lines
Quoted 6 lines
Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- sendxfs_infooutput for the filesystem in question;
- dump the log (xfs_logprint-C) and send it to us.Sure. But in what order should I do all that?
[SNIPPED]
---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/
95% of the programmers consider 95% of the code they did not write, in the bottom 5%.
Quoted 5 lines · Shlomi Fish
Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.
Quoted 3 lines
Quoted 3 lines
- get sysrq-t information for all hung processes, esp. mount;
- sendxfs_infooutput for the filesystem in question;
- dump the log (xfs_logprint-C) and send it to us.
xfs_logprint -C /dev/foo against the unmounted device.
sysrq-t when it's hung, after you try to mount
xfs_info /mnt/point when it's mounted.
Well, you can't mount, because recovery hangs, so,
mount -o ro,norecovery /dev/foo /mnt/point; xfs_info /mnt/point
-Eric
Quoted 13 lines · Eric Sandeen
Quoted 7 lines · Shlomi Fish
Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.
Quoted 3 lines
Quoted 3 lines
- get sysrq-t information for all hung processes, esp. mount;
- sendxfs_infooutput for the filesystem in question;
- dump the log (xfs_logprint-C) and send it to us.
xfs_logprint-C/dev/fooagainst the unmounted device.sysrq-t when it's hung, after you try to mount
Quoted 7 lines
xfs_info/mnt/pointwhen it's mounted.Well, you can't mount, because recovery hangs, so,
mount -o ro,norecovery
/dev/foo/mnt/point;xfs_info/mnt/point-Eric
Thanks!
I'll save this message to a file, put it on different partitions and print it in case this problem repeats itself.
Regards,
Shlomi Fish
---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/
95% of the programmers consider 95% of the code they did not write, in the bottom 5%.