ls hangs filesystem

DISCUSSION Jim 3 participants 12 messages Oct 18, 2011 Btrfs file system

Share & tools

Jim

author

2011-10-18 13:25 UTC #0 lore raw

Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice. Jim

[root@btrfs ~]# tail -20 /var/log/messages

Oct 18 05:06:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0
Oct 18 05:06:36 btrfs kernel: [<ffffffff814b19ab>] system_call_fastpath+0x16/0x1b
Oct 18 05:08:36 btrfs kernel: INFO: task ls:7100 blocked for more than 120 seconds.
Oct 18 05:08:36 btrfs kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 05:08:36 btrfs kernel: ls              D ffff88080bc843d8     0  7100   7047 0x00000084
Oct 18 05:08:36 btrfs kernel: ffff8804f526dd68 0000000000000086 ffff8804f526dd18 ffffffff00000000
Oct 18 05:08:36 btrfs kernel: ffff88080bc84000 0000000000011900 ffff8804f526dfd8 ffff8804f526c010
Oct 18 05:08:36 btrfs kernel: ffff8804f526dfd8 0000000000011900 ffff88080f64d7f0 ffff88080bc84000
Oct 18 05:08:36 btrfs kernel: Call Trace:
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a0f45>] wait_current_trans+0xa5/0x110 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffff8106c0e0>] ? wake_up_bit+0x40/0x40
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2540>] start_transaction+0x1e0/0x2b0 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2665>] btrfs_join_transaction+0x15/0x20 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01ad360>] btrfs_dirty_inode+0x50/0x160 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffff8115631f>] __mark_inode_dirty+0x3f/0x200
Oct 18 05:08:36 btrfs kernel: [<ffffffff81148bf5>] touch_atime+0x115/0x150
Oct 18 05:08:36 btrfs kernel: [<ffffffff81142520>] ? sys_ioctl+0xb0/0xb0
Oct 18 05:08:36 btrfs kernel: [<ffffffff811426ce>] vfs_readdir+0xce/0xe0
Oct 18 05:08:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0
Oct 18 05:08:36 btrfs kernel: [<ffffffff814b19ab>] system_call_fastpath+0x16/0x1b

Josef Bacik

2011-10-18 13:31 UTC #1 lore raw

Quoted 4 lines · Jim

Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux

Quoted 8 lines

shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice.

Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks,

Josef

Jim

author

2011-10-18 13:47 UTC #2 lore raw

I am on kernel 3.1.0-rc4. I am ssh'd into a remote machine so have no access to keyboard, any other way to get the info you need? Thanks again. Jim

Quoted 17 lines · Josef Bacik

Quoted 4 lines · Jim
Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux
Quoted 8 lines
shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice.
Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks,
Josef

Sander

2011-10-18 13:49 UTC #3 lore raw

Jim wrote (ao):

Quoted 2 lines

I am on kernel 3.1.0-rc4. I am ssh'd into a remote machine so have no access to keyboard, any other way to get the info you need?

echo 'w' > /proc/sysrq-trigger

Quoted 22 lines · Josef Bacik

Quoted 13 lines · Jim
Quoted 4 lines
Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux
Quoted 8 lines
shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice.
Quoted 4 lines
Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks,
Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

Humilis IT Services and Solutions
http://www.humilis.net

Jim

author

2011-10-18 13:50 UTC #4 lore raw

Thank you, I should have thought of that. I will get back as soon as I have info. Jim

Quoted 28 lines · Sander

Jim wrote (ao):
Quoted 2 lines
I am on kernel 3.1.0-rc4. I am ssh'd into a remote machine so have no access to keyboard, any other way to get the info you need?
echo 'w'> /proc/sysrq-trigger
Quoted 22 lines · Josef Bacik
Quoted 13 lines · Jim
Quoted 4 lines
Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux
Quoted 8 lines
shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice.
Quoted 4 lines
Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks,
Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jim

author

2011-10-18 14:03 UTC #5 lore raw

In reply to Sander

I am getting no output from echo 'w' > /proc/sysrq-trigger. Ls is hung again so I can try any thing else you suggest to get data. Jim

Quoted 28 lines · Sander

Jim wrote (ao):
Quoted 2 lines
I am on kernel 3.1.0-rc4. I am ssh'd into a remote machine so have no access to keyboard, any other way to get the info you need?
echo 'w'> /proc/sysrq-trigger
Quoted 22 lines · Josef Bacik
Quoted 13 lines · Jim
Quoted 4 lines
Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux
Quoted 8 lines
shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can't umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice.
Quoted 4 lines
Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks,
Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Josef Bacik

2011-10-18 14:04 UTC #6 lore raw

Quoted 2 lines · Jim

I am getting no output from echo 'w' > /proc/sysrq-trigger. Ls is hung again so I can try any thing else you suggest to get data.

echo 1> /proc/sys/kernel/sysrq

and then try doing the echo w thing. Thanks,

Josef

Jim

author

2011-10-18 14:08 UTC #7 lore raw

Yes, I had thought of that as well but still no output. Ls has been hung for about 30 min now and shows as D+ in ps. Do you think it would get us any information if I forced a reboot and redid the entire sequence running sysrq-trigger right after starting ls? Jim

Quoted 10 lines · Josef Bacik

Quoted 2 lines · Jim
I am getting no output from echo 'w'> /proc/sysrq-trigger. Ls is hung again so I can try any thing else you suggest to get data.
Do
echo 1> /proc/sys/kernel/sysrq
and then try doing the echo w thing. Thanks,
Josef

Josef Bacik

2011-10-18 14:09 UTC #8 lore raw

Quoted 4 lines · Jim

Yes, I had thought of that as well but still no output. Ls has been hung for about 30 min now and shows as D+ in ps. Do you think it would get us any information if I forced a reboot and redid the entire sequence running sysrq-trigger right after starting ls?

Maybe sysrq+t will get it? Thanks,

Josef

Jim

author

2011-10-18 14:10 UTC #9 lore raw

Thank you both (Sander and Josef) I can forget anything so all reminders are appreciated. I will try again and also use the t argument. Jim

Quoted 8 lines · Josef Bacik

Quoted 4 lines · Jim
Yes, I had thought of that as well but still no output. Ls has been hung for about 30 min now and shows as D+ in ps. Do you think it would get us any information if I forced a reboot and redid the entire sequence running sysrq-trigger right after starting ls?
Maybe sysrq+t will get it? Thanks,
Josef

Jim

author

2011-10-18 14:26 UTC #10 lore raw

In reply to Josef Bacik

Thank you all for your prompt responses. I forced a reboot, remounted the fs and now cannot recreate the behavior. Ls appears to be working as expected. If it hangs later today I will get all the info I can and repost under this subject. Thank you again for your assistance. Jim

Quoted 8 lines · Josef Bacik

Quoted 4 lines · Jim
Yes, I had thought of that as well but still no output. Ls has been hung for about 30 min now and shows as D+ in ps. Do you think it would get us any information if I forced a reboot and redid the entire sequence running sysrq-trigger right after starting ls?
Maybe sysrq+t will get it? Thanks,
Josef

Sander

2011-10-18 14:08 UTC #11 lore raw

In reply to Josef Bacik

Josef Bacik wrote (ao):

Quoted 9 lines · Jim

Quoted 2 lines
I am getting no output from echo 'w' > /proc/sysrq-trigger. Ls is hung again so I can try any thing else you suggest to get data.
Do
echo 1> /proc/sys/kernel/sysrq
and then try doing the echo w thing. Thanks,

Jim, sorry if I'm stating the obvious here, but the output will be in dmesg (and syslog if possible).

Sander

Signature · 2 lines

Humilis IT Services and Solutions
http://www.humilis.net