Skip to contentSkip to conversation

XFS Mount Hangs the Partition (on latest kernel + many old 2.6.x ones)

DISCUSSION Shlomi Fish 3 participants 6 messages Dec 7, 2005 – Dec 15, 2005

lore
Share & tools
lore
#0 lore raw

Hi all!

(Please CC me on replies)

I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "xfs_check" or "xfs_repair") to hang too. However, running xfs_check
or xfs_repair before the first mount (after a reboot) worked, and eventually resolved this problem.

I blogged about it (relatively incoherently) here:

http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply

It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.

Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).

After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that xfs_check and xfs_repair both hanged up as well. After a reboot, I tried running xfs_check right away on that partition and it worked. So I ran xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.

I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by xfs_check and xfs_repair to fail as well.

I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.

Regards,

Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the bottom 5%.

#1 lore raw
Quoted 1 line · Shlomi Fish

Hi all!

Hi there,

Quoted 10 lines

(Please CC me on replies)

I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "xfs_check" or "xfs_repair") to hang too. However, running xfs_check
or xfs_repair before the first mount (after a reboot) worked, and eventually resolved this problem.

Quoted 4 lines

I blogged about it (relatively incoherently) here:

http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply

Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.

Quoted 5 lines

It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.

Quoted 8 lines

Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).

After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that xfs_check and xfs_repair both hanged up as well. After a reboot, I tried running xfs_check

This was probably caused by the block device being held open exclusively by the stuck mount process.

Quoted 11 lines

right away on that partition and it worked. So I ran xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.

I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by xfs_check and xfs_repair to fail as well.

Quoted 3 lines

I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.

Thats unlikely to help now - repair will have wiped your log clean, so all evidence of the problem will be gone.

cheers.

Signature · 1 line
Nathan
#2 lore raw
Quoted 4 lines · Nathan Scott
Quoted 1 line · Shlomi Fish

Hi all!

Hi there,

Hi Mr. Scott (and all)!

Quoted 16 lines
Quoted 11 lines

(Please CC me on replies)

I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "xfs_check" or "xfs_repair") to hang too. However, running xfs_check
or xfs_repair before the first mount (after a reboot) worked, and eventually resolved this problem.

Quoted 4 lines

I blogged about it (relatively incoherently) here:

http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply

Quoted 6 lines

Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.

Sure. But in what order should I do all that?

Quoted 12 lines
Quoted 12 lines

It all happened after I detected some problems on my Mandriva 2006 system (that was using kernel 2.4.15-rc2 from Linus), and then rebooted twice, thinking something went wrong. Then a loadlin-booted kernel was unable to load the kernel.

Knoppix ran fine, but it also hang up on attempting to mount the XFS partition. It used a much older kernel. I then tried to boot Kubuntu (which was on another XFS partition on the same disk) and it booted fine. Still, it was unable to mount the partition. (It too had an older kernel).

Quoted 7 lines
Quoted 4 lines

After I compiled a 2.6.14.3 kernel, and booted Kubuntu with it, it again could not mount the XFS partition, and after doing that xfs_check and xfs_repair both hanged up as well. After a reboot, I tried running xfs_check

This was probably caused by the block device being held open exclusively by the stuck mount process.

I see.

Quoted 14 lines
Quoted 14 lines

right away on that partition and it worked. So I ran xfs_repair, and after it finished, tried to mount the partition it worked. Then Mandriva booted fine.

I did not had any problems since then (I have an uptime of 11 days now using kernel 2.6.14.3), and so it doesn't seem like a hard disk problem. Something using kernel 2.6.15-rc2 caused the XFS partition to become defected, and worse - something in all the kernels starting from that of the first official Kubuntu release, (or the Knoppix I had), caused an attempt to mount the Mandriva partition to hang the process, and subsequent accesses to the partition by xfs_check and xfs_repair to fail as well.

Quoted 6 lines
Quoted 3 lines

I can no longer reproduce the problem, but it might be worth going over the code. If it helps I can privately send a dump of the first 131,072,000 bytes of the XFS partition to someone trustworthy.

Thats unlikely to help now - repair will have wiped your log clean, so all evidence of the problem will be gone.

Yes. I thought one can try looking in the XFS mounting code for possible bugs.

Regards,

Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the bottom 5%.

#3 lore raw

Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.

Regards,

Shlomi Fish

Quoted 23 lines · Shlomi Fish
Quoted 4 lines · Nathan Scott
Quoted 1 line · Shlomi Fish

Hi all!

Hi there,

Hi Mr. Scott (and all)!

Quoted 15 lines
Quoted 14 lines

(Please CC me on replies)

I encountered a problem with the Linux kernel handling of XFS, in which attempting to mount a certain XFS partition (but not a different one on the same hard-disk) caused the mount process to hang, and all other XFS-aware apps (like "xfs_check" or "xfs_repair") to hang too. However, running xfs_check
or xfs_repair before the first mount (after a reboot) worked, and eventually resolved this problem.

I blogged about it (relatively incoherently) here:

http://www.livejournal.com/~shlomif/7182.html?mode=reply
http://www.livejournal.com/~shlomif/7547.html?mode=reply

Quoted 8 lines
Quoted 6 lines

Unfortunately there's not much information here in your mail or there that would help us to analyse this further. If you see this behaviour again could you:
- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.

Sure. But in what order should I do all that?

[SNIPPED]

---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the bottom 5%.

#4 lore raw
Quoted 5 lines · Shlomi Fish

Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.

Quoted 3 lines
Quoted 3 lines

- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.

xfs_logprint -C /dev/foo against the unmounted device.

sysrq-t when it's hung, after you try to mount

xfs_info /mnt/point when it's mounted.

Well, you can't mount, because recovery hangs, so,

mount -o ro,norecovery /dev/foo /mnt/point; xfs_info /mnt/point

-Eric

#5 lore raw
Quoted 13 lines · Eric Sandeen
Quoted 7 lines · Shlomi Fish

Replying to myself, I'd like to ask for a response to my previous message, especially the order in which the operations Mr. Scott mentioned need to be performed.

Quoted 3 lines
Quoted 3 lines

- get sysrq-t information for all hung processes, esp. mount;
- send xfs_info output for the filesystem in question;
- dump the log (xfs_logprint -C) and send it to us.

xfs_logprint -C /dev/foo against the unmounted device.

sysrq-t when it's hung, after you try to mount

Quoted 7 lines

xfs_info /mnt/point when it's mounted.

Well, you can't mount, because recovery hangs, so,

mount -o ro,norecovery /dev/foo /mnt/point; xfs_info /mnt/point

-Eric

Thanks!

I'll save this message to a file, put it on different partitions and print it in case this problem repeats itself.

Regards,

Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish [email protected]
Homepage: http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the bottom 5%.